Does Time Machine back up the files stored in an encrypted .dmg?

[qqurioustiger8945]

Hey everyone,

In order to hide my sensitive documents I've created an encrypted .dmg with Disk Utility and I've put them all in. Those documents exist nowhere else on my hard drive.

When I use Time Machine to backup my HD, does TM automatically detect changes within that encrypted .dmg and thus backs up the files stored in it?

Or is it that this .dmg is forever perceived as one file and no matter what changes happen inside it, TM has no way of knowing?

Thank you very much.

** extra question: What if I stopped using an encrypted .dmg through Disk Utility and started using an encypted VeraCrypt container instead? Or what if I moved from TM to the known application Carbon Copy Cloner? Would any of that make any difference?

 

[Weaselboy]

TM will see that the DMG has changed and back it up again.

 

[qqurioustiger8945]

TM will see that the DMG has changed and back it up again.

Ok, so as I modify the documents inside the .dmg, it's like every time there's a different version of this .dmg file, so TM backs up the entire .dmg with its last version and replaces the old one with the new one, right?

NOT just the specific documents that were modified in it, as if this .dmg was a folder instead.

Did I get it right? Would you know if that is also how Carbon Copy Cloner treats encrypted .dmg's?

Thank you.

 

[Weaselboy]

Ok, so as I modify the documents inside the .dmg, it's like every time there's a different version of this .dmg file, so TM backs up the entire .dmg with its last version and replaces the old one with the new one, right?

NOT just the specific documents that were modified in it, as if this .dmg was a folder instead.

Did I get it right? Would you know if that is also how Carbon Copy Cloner treats encrypted .dmg's?

Thank you.

Yes... each time you make a change TM is going to backup the whole DMG again. Not a big deal if it is a few files, but if you have a 1GB DMG, I don't think this will be a very good idea.

CCC is going to work the same way.

Why don't you just turn on FileVault and encrypt the whole drive and you won't have this issue?

 

[chabig]

Ok, so as I modify the documents inside the .dmg, it's like every time there's a different version of this .dmg file, so TM backs up the entire .dmg with its last version and replaces the old one with the new one, right?

Not exactly. Time Machine backs up the new version of the file (the .dmg) in addition to all of the previous backups. It doesn’t replace files. It keeps them all.

NOT just the specific documents that were modified in it, as if this .dmg was a folder instead.

You can think of the dmg as an opaque folder if it helps. When anything inside changes, the “folder” is now different, and will be backed up.

Did I get it right? Would you know if that is also how Carbon Copy Cloner treats encrypted .dmg's?

Yes. CCC sees a newly changed file and copies it.

Thank you.

 

[chown33]

Yes... each time you make a change TM is going to backup the whole DMG again. Not a big deal if it is a few files, but if you have a 1GB DMG, I don't think this will be a very good idea.

CCC is going to work the same way.

Why don't you just turn on FileVault and encrypt the whole drive and you won't have this issue?

A sparsebundle will solve the "1GB DMG" problem. A sparseimage is fundamentally a directory with multiple 8MB files in it (bands). Each file holds the data for one 8MB section (band) of the represented disk. If data files on the represented disk don't change, then the band file doesn't change.

If you look inside the bundle (right-click, Show Package Contents), you can see the band files, and then when you write files to the image, you can observe which band files change their modification data, and which don't.

https://en.wikipedia.org/wiki/Sparse_image

 

[Weaselboy]

A sparseimage is fundamentally a directory with multiple 8MB files in it (bands).

Thanks.... I'm familiar with how the bands work, but was not sure if TM would recognize that and just copy over the changed bands?

 

[chabig]

Thanks.... I'm familiar with how the bands work, but was not sure if TM would recognize that and just copy over the changed bands?

I don't think TM does. Dropbox, however, does just copy the changed bands. That's a really nice aspect of Dropbox.

 

[chown33]

I don't think TM does. Dropbox, however, does just copy the changed bands. That's a really nice aspect of Dropbox.

I'd think TM would only copy files (bands) that changed. I don't think TM has any knowledge of sparse bundles; it should just be a dir of files.

One thing that might cause TM to copy an otherwise unchanged band would be a change in metadata or xattrs.

It'd be worth setting up a test to see what happens. The hard-link count (ls -l in Terminal) of the bands in the TM backup will tell you whether TM made a copy or a simple hard-link.

 

[qqurioustiger8945]

A sparsebundle will solve the "1GB DMG" problem. A sparseimage is fundamentally a directory with multiple 8MB files in it (bands). Each file holds the data for one 8MB section (band) of the represented disk. If data files on the represented disk don't change, then the band file doesn't change.

If you look inside the bundle (right-click, Show Package Contents), you can see the band files, and then when you write files to the image, you can observe which band files change their modification data, and which don't.

https://en.wikipedia.org/wiki/Sparse_image

Thank you for your response.

It's the first time I hear about sparsebundle. Is there anything I need to do in order to set it up and transfer my sensitive documents in it?

 

[chown33]

Thank you for your response.

It's the first time I hear about sparsebundle. Is there anything I need to do in order to set it up and transfer my sensitive documents in it?

I've never done anything special, if that's what you're asking.

You create it like any DMG. It should be an option when you're creating it. So the thing you do is choose a sparse bundle. Other options, such as encrypted or size, should be the same.

 

[qqurioustiger8945]

I've never done anything special, if that's what you're asking.

You create it like any DMG. It should be an option when you're creating it. So the thing you do is choose a sparse bundle. Other options, such as encrypted or size, should be the same.

Got it.


Thank you all by the way.

 

[qqurioustiger8945]

I've never done anything special, if that's what you're asking.

You create it like any DMG. It should be an option when you're creating it. So the thing you do is choose a sparse bundle. Other options, such as encrypted or size, should be the same.

Right now, I keep my sensitive documents in an encrypted DMG, but I'll move them all in a sparsebundle instead, because apparently it saves me a lot of time due to it backing up incrementally.

As far as the "technology" behind sparsebundles is concerned, do I have to worry about file corruption more (or less) than DMG's? Meaning, if I'm just as careful (after moving to sparsebundle from DMG), do I have a higher (or lower) risk to encounter corrupted files?

Also, when a sparsebundle fills up with files and reaches its limit in size, is it advised to make a new one and move the files in it, or should I just run the Terminal command (if I'm not mistaken it's hdiutil compact /Volumes/*name...*/*name...*.sparsebundle) which reclaims the unused space for me?

As you can probably already see, first and foremost, my goal is to avoid encountering corrupted files. If using a sparsebundle instead of a DMG isn't as safe, I could definitely give up the advantage of the time I save when backing up.

Thank you.

 

[chown33]

Right now, I keep my sensitive documents in an encrypted DMG, but I'll move them all in a sparsebundle instead, because apparently it saves me a lot of time due to it backing up incrementally.

As far as the "technology" behind sparsebundles is concerned, do I have to worry about file corruption more (or less) than DMG's? Meaning, if I'm just as careful (after moving to sparsebundle from DMG), do I have a higher (or lower) risk to encounter corrupted files?

Also, when a sparsebundle fills up with files and reaches its limit in size, is it advised to make a new one and move the files in it, or should I just run the Terminal command (if I'm not mistaken it's hdiutil compact /Volumes/*name...*/*name...*.sparsebundle) which reclaims the unused space for me?

As you can probably already see, first and foremost, my goal is to avoid encountering corrupted files. If using a sparsebundle instead of a DMG isn't as safe, I could definitely give up the advantage of the time I save when backing up.

Thank you.

What does being careful actually mean? What actions would you call careful? What would you consider careless?

If you have examples of careful vs. careless, please post them, and I can think about what the consequences of those actions might be. But without knowing exactly what you mean, I can't say one way or the other.

The "technologY" is pretty simple. A dmg (or sparseimage) is a single file. A sparsebundle is a directory with multiple files. If you know anything about filesystems, and HFS in particular, then you'd know that corruption can occur for various reasons, at various points in the writing of various blocks of data.

The primary factor in determining reliability will be the reliability of the media itself, i.e. the SSD, rotating HD, SD card, USB flash drive, or whatever the disk image is stored on. If the media is unreliable or fails, then nothing in the dmg, sparseimage, or sparsebundle can correct for that.

One advantage that a sparsebundle has over a sparseimage arises from the bands. Because each is a separate file, holding some part of the overall "disk", if a band holds data that doesn't change, then that band file isn't modified, even if other parts of the "disk" are. For example, if you don't change certain files on the sparsebundle, then the bands that store the actual data for those files isn't modified. This makes it less likely that existing unchanged files will be corrupted if something goes wrong elsewhere on the "disk". In other words, existing unchanged data is less likely to be corrupted if some new data ends up getting corrupted. Of course, that also depends on what kind of corruption you're concerned about. Directory corruption can damage the ability to locate or access existing files, so even if the file data is undamaged, the directory that holds all the location info could be damaged. I can't see a reason to think that sparsebundle is better (less corruptible) than dmg or sparseimage in that regard.

If you do things you shouldn't, like unplug removable media without ejecting the volume first, then all forms of disk image are susceptible to corruption. I'm not sure how to assess one format's reliability over the other.

One thing that a dmg or sparseimage will give you that a sparsebundle won't is singular integrity. That is, if Time Machine is backing up a complete dmg or sparseimage, then the complete file is backed up. It's a single file (singular), and the entire file is consistent (integrity).

Sadly, "integrity" doesn't tell you whether the disk image file is corrupted or not. It could be that something happened in the past that damaged a directory, and unless you run Disk Utility on the disk image, you won't know it. So some past corruption could occur, and because each backup copies the whole disk image file, you keep carrying that corruption forward. Since sparsebundle bands work differently (they don't have singular integrity), you might get different kinds of failures if directory corruption is carried forward.

As a general rule, I'd keep multiple backups of whatever was important. That means multiple media, as in an SD card, a rotating HD, and maybe a USB flash drive or a second SD card. I'd also be verifying the multiple backups, so if there is any data loss, it becomes apparent before you try relying on the data.


If that seems like a lot of verbiage, it's because you asked a somewhat unclear question, and any answer will be complex because it depends on multiple tradeoffs. Plus there's the tradeoffs involved with the single huge dmg vs. the distinct band-files of the sparsebundle. The size benefits might outweigh the risks; only you can judge that.

 

[qqurioustiger8945]

What does being careful actually mean? What actions would you call careful? What would you consider careless?

If you have examples of careful vs. careless, please post them, and I can think about what the consequences of those actions might be. But without knowing exactly what you mean, I can't say one way or the other.

The "technologY" is pretty simple. A dmg (or sparseimage) is a single file. A sparsebundle is a directory with multiple files. If you know anything about filesystems, and HFS in particular, then you'd know that corruption can occur for various reasons, at various points in the writing of various blocks of data.

The primary factor in determining reliability will be the reliability of the media itself, i.e. the SSD, rotating HD, SD card, USB flash drive, or whatever the disk image is stored on. If the media is unreliable or fails, then nothing in the dmg, sparseimage, or sparsebundle can correct for that.

One advantage that a sparsebundle has over a sparseimage arises from the bands. Because each is a separate file, holding some part of the overall "disk", if a band holds data that doesn't change, then that band file isn't modified, even if other parts of the "disk" are. For example, if you don't change certain files on the sparsebundle, then the bands that store the actual data for those files isn't modified. This makes it less likely that existing unchanged files will be corrupted if something goes wrong elsewhere on the "disk". In other words, existing unchanged data is less likely to be corrupted if some new data ends up getting corrupted. Of course, that also depends on what kind of corruption you're concerned about. Directory corruption can damage the ability to locate or access existing files, so even if the file data is undamaged, the directory that holds all the location info could be damaged. I can't see a reason to think that sparsebundle is better (less corruptible) than dmg or sparseimage in that regard.

If you do things you shouldn't, like unplug removable media without ejecting the volume first, then all forms of disk image are susceptible to corruption. I'm not sure how to assess one format's reliability over the other.

One thing that a dmg or sparseimage will give you that a sparsebundle won't is singular integrity. That is, if Time Machine is backing up a complete dmg or sparseimage, then the complete file is backed up. It's a single file (singular), and the entire file is consistent (integrity).

Sadly, "integrity" doesn't tell you whether the disk image file is corrupted or not. It could be that something happened in the past that damaged a directory, and unless you run Disk Utility on the disk image, you won't know it. So some past corruption could occur, and because each backup copies the whole disk image file, you keep carrying that corruption forward. Since sparsebundle bands work differently (they don't have singular integrity), you might get different kinds of failures if directory corruption is carried forward.

As a general rule, I'd keep multiple backups of whatever was important. That means multiple media, as in an SD card, a rotating HD, and maybe a USB flash drive or a second SD card. I'd also be verifying the multiple backups, so if there is any data loss, it becomes apparent before you try relying on the data.


If that seems like a lot of verbiage, it's because you asked a somewhat unclear question, and any answer will be complex because it depends on multiple tradeoffs. Plus there's the tradeoffs involved with the single huge dmg vs. the distinct band-files of the sparsebundle. The size benefits might outweigh the risks; only you can judge that.

You're right, I apologize. "Just as careful" was probably not the best way to phrase myself. Really what I meant to say is that my interaction with my computer does not alter, before or after transitioning to sparsebundle from DMG; that the only difference that occurs is that files existed in a DMG in the past and now they exist in a sparsebundle, and me as a user, I don't change my routine and the way that I use or interact with my computer.

This concern of mine stems from a conversation I had with an acquaintance whose opinion was that corrupted files is a more common occurrence in sparsebundles when compared to DMG. It was later confirmed by another one, so I thought about asking on MacRumors to see if that's the consensus, or was it just isolated incidents with those two folks.

Unfortunately, the worst type of file corruption is the one that goes unnoticed; if an excel file (whether in a DMG or a sparsebundle) I use only once a year became corrupted at some point and I find out about it next year (by then my TM backups have overwritten themselves and the oldest version of that excel file in them is the corrupted one), then there's not much I can do to retrieve it.

If you do things you shouldn't, like unplug removable media without ejecting the volume first, then all forms of disk image are susceptible to corruption. I'm not sure how to assess one format's reliability over the other.

This is probably a silly question, but would I also have to eject the actual sparsebundle that is mounted? Up until now, I never paid attention to ejecting my mounted DMG's before shutting down, logging out, etc. Is it the same with sparsebundle, or should I make sure they are ejected first?

By the way thank you for the time you spent to write such a detailed answer, I appreciate it.

 

[chown33]

You're right, I apologize. "Just as careful" was probably not the best way to phrase myself. Really what I meant to say is that my interaction with my computer does not alter, before or after transitioning to sparsebundle from DMG; that the only difference that occurs is that files existed in a DMG in the past and now they exist in a sparsebundle, and me as a user, I don't change my routine and the way that I use or interact with my computer.

You didn't actually say what actions you think are careful or careless. It could be that what you're doing right now is risky, or something that I would consider careless. (I'm not saying it is, I'm giving a hypothetical.) Since I don't know what you're doing now, I can't say whether it's risky or not.

This concern of mine stems from a conversation I had with an acquaintance whose opinion was that corrupted files is a more common occurrence in sparsebundles when compared to DMG. It was later confirmed by another one, so I thought about asking on MacRumors to see if that's the consensus, or was it just isolated incidents with those two folks.

I'd have to know what the reason was for those people to say that. It could be well-reasoned, based on lots of direct experience in a variety of circumstances, and on theoretical understanding of how HFS+ works. Or it could just be a couple of anecdotal data points. I have no way to know which one it is.

Did you also ask a whole lot of people who then gave you positive answers? I.e. did you do an actual survey of some kind? Because if you didn't, then you've got selection bias. That means you've only recorded negative results (sparsebundle failures) because you haven't been systematically collecting all results and then sorting them into positive and negative counts.

In my experience, I've seen no difference between different disk image formats when it comes to risk. Unfortunately, that's just another anecdotal data point, because I haven't told you how many hours I've been using them, or what kinds of reliability tests I might have done.

Unfortunately, the worst type of file corruption is the one that goes unnoticed; if an excel file (whether in a DMG or a sparsebundle) I use only once a year became corrupted at some point and I find out about it next year (by then my TM backups have overwritten themselves and the oldest version of that excel file in them is the corrupted one), then there's not much I can do to retrieve it.

That's why you don't put all your eggs in one basket. Make multiple copies on several different media. Even a Time Machine backup disk can fail, and if that's your only backup, then the reliability of sparsebundle vs. dmg is irrelevant, because you were sunk by media reliability shortcomings.

You should manage your backups differently, depending on how long you need to keep a file. If you have a file that's used once a year, consider putting it on long-lived media, possibly even write-once archival media. You should also use multiple media, so if any single one fails, you have extras. To dispose of obsolete copies, use a shredder that can handle CDs or DVD's.

If it's important, you should systematically test for corruption more often than once a year. The only way I know of to check an Excel file for validity is to open it in Excel and see if it works. You could also compare a backup copy B with a known-good file A, but unless you have an independent way of confirming that A is actually correct and uncorrupted, then all that does is tell you if there's a difference. It might be that B is correct and A has been corrupted. There are ways of discovering those kinds of problems (keep hashes of files on separate media), but I don't know if there's an easy-to-use app for that.

This is probably a silly question, but would I also have to eject the actual sparsebundle that is mounted? Up until now, I never paid attention to ejecting my mounted DMG's before shutting down, logging out, etc. Is it the same with sparsebundle, or should I make sure they are ejected first?

The system will automatically eject all media (including all disk images, of any format) before it shuts down, for a NORMAL shutdown. It MIGHT NOT do this for a forced shutdown.

I don't recall if it ejects everything if you only logout. You could check by logging out, then logging in and seeing if the disk image is still mounted. Make sure you don't have a Login Item that opens (mounts) the disk image.

I know for certain that a sleep DOES NOT eject media, either disk images, network shares, or external drives. I think there's a security option that might blank out the FileVault key on sleep, but I don't remember if that also writes all pending updates before the key is blanked. As a sane design it should, but sadly, bugs exist.

 

[qqurioustiger8945]

I'd have to know what the reason was for those people to say that. It could be well-reasoned, based on lots of direct experience in a variety of circumstances, and on theoretical understanding of how HFS+ works. Or it could just be a couple of anecdotal data points. I have no way to know which one it is.

It was just a casual conversation with a couple of colleagues who are a little bit more tech inclined than I am. They literally just said that in their experience, they've encountered more file corruption while using sparsebundle, and not so much with DMG. And that if I don't mind the extra time it takes for TM to backup, I should go with DMG. They may (or not be) mistaken, so I figured I should check with MacRumors.

There are ways of discovering those kinds of problems (keep hashes of files on separate media), but I don't know if there's an easy-to-use app for that.

You mean, like MD5? If yes, I've been looking for an app where I just input an entire directory and checks all files in it. So far, all I've found is a couple of apps where you test for selected files, but not folders of files, or folders with folders and so forth. If I could find a way to automatically MD5-check every file on my Mac and its equivalent on my backup drive to compare which files have been modified, that could probably limit the occurrence of corrupted files going by unnoticed.

The system will automatically eject all media (including all disk images, of any format) before it shuts down, for a NORMAL shutdown.

I don't recall if it ejects everything if you only logout. You could check by logging out, then logging in and seeing if the disk image is still mounted.

It does.

It MIGHT NOT do this for a forced shutdown.

In that case, would I risk file corruption inside a mounted DMG (or sparsebundle)?

 

[chown33]

You mean, like MD5?

I'd probably use one of the more recent SHA versions.

In that case, would I risk file corruption inside a mounted DMG (or sparsebundle)?

Yes.

You'd also risk corruption on every medium (disk, SD card, etc.) that's mounted read/write.

There's no risk to media that's mounted read-only, such as an SD card with its write-protect slider in the no-write position.

 

[Weaselboy]

You mean, like MD5? If yes, I've been looking for an app where I just input an entire directory and checks all files in it. So far, all I've found is a couple of apps where you test for selected files, but not folders of files, or folders with folders and so forth. If I could find a way to automatically MD5-check every file on my Mac and its equivalent on my backup drive to compare which files have been modified, that could probably limit the occurrence of corrupted files going by unnoticed.

The cloning app Carbon Copy Cloner has a feature that will do that for you.

Here is a copy pasta from the CCC help.

Find and replace corrupted files, "Backup Health Check"

CCC normally uses file size and modification date to determine whether a file should be copied. With this option, CCC will calculate an MD5 checksum of every file on the source and every corresponding file on the destination. CCC then uses these MD5 checksums to determine if a file should be copied. This option will increase your backup time, but it will expose any corrupted files within your backup set on the source and destination. This is a reliable method of verifying that the files that have been copied to your destination volume actually match the contents of the files on the source volume.

Media failures occur on nearly every hard drive at some point in the hard drive's life. These errors affect your data randomly, and go undetected until an attempt is made to read data from the failed sector of media. If a file has not been modified since a previous (successful) backup, CCC will not ordinarily attempt to read every byte of that file's content. As a result, it is possible for a corrupted file to go unnoticed on your source or destination volume. Obviously this is a concern if the file is important, and one day you actually need to recover the contents of that file.

Frequent use of the checksum calculation option is unnecessary and may be a burden upon your productivity, so CCC offers weekly and monthly options to limit how frequently the checksumming occurs.