I'm trying to make a copy of a TimeMachine backup. I have it stored on a 2 TB external HD. When I go to copy just a portion of the dated backup folders to a new HD it is over 3 TB of files. Which, this isn't possible because the disk isn't that large. Any ideas what I'm doing wrong.
Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Copying TimeMachine backup? Unexplainable large file size.
- Thread starter Qwerty11
- Start date
- Sort by reaction score
How are you copying? With the Finder? I have used Carbon Copy Cloner to copy a large time machine disk without issues.
Time Machine uses a file system feature called "hard links". It's when the same physical file on the disk has many different directory entries. Whatever tool you are using to copy the data would need special code to deal with this scenario since it isn't a very commonly used feature. Finder probably doesn't know how to deal with that. I've not used CCC, but it probably does. Another option is to use rsync on the command-line in a Terminal shell. That tool has the ability to copy files with hard links properly. Depends on what you are comfortable using. CCC is probably the easiest. You could also use the 'Restore' feature in Disk Utility. Tell Disk Utility to "Restore" the source disk onto the Destination Disk. I bet that would work properly.
Also, you'd need to be copying to a disk that has a filesystem that knows how to deal with hard links. APFS and HFS+ can use hard links. ExFAT and NTFS cannot.
Also, you'd need to be copying to a disk that has a filesystem that knows how to deal with hard links. APFS and HFS+ can use hard links. ExFAT and NTFS cannot.
To be fair, I was just cloning an entire time machine disk, not messing with the folders, not sure if CCC can do that. Why are you doing this? Are you trying to reduce the size of your time machine backup? I'd be careful about doing that since time machine normally does its own "housecleaning".
Finder does preserve hard links (just tested it: seemed to recall this as that's how I moved my Pictures library to external drive years ago).
cp command does not (verified that as well). man 1 cp:
Believe rsync has option to preserve hard links as well.
But, yeah, if HFS+ Time Machine, hard links, and will replicate each hard link of the file as a separate instance of the file. APFS Time Machine, no idea, as I've never tried to backup the TM drive and APFS is using snapshots now.
This is definitely possible when the backup disk is formatted as APFS -- as it will be if the Time Machine backup was started under a recent version of macOS. In this case, one must consider the new APFS features of sparse files and clone files. There are many relevant posts at https://eclecticlight.co. about these new storage schemes and how Time Machine deals with them.
Sparse files may take much less disk space than their "actual" size. A program can create a 10GB file for example, but only "seek" to various spots within the file to write, say 200 MB of data. On disk, it would only use up about 200MB -- not even close to its potential 10GB size. The above site indicates that it this is not uncommon for system files and databases.
Clone files: say you use Finder to make an exact copy of a 5GB file. APFS will show two separate files (inodes), but on disk, only 5GB will be used up! Now if you modify one of them, say by changing 100MB, only the changed data is written to disk. So the two files together (which look in Finder to total 10GB), will only use 5.1GB of disk space!
Here's the thing about these special file types: Time Machine is able to copy them from your source volume to an APFS-formatted backup volume while retaining these special properties. However, (at least at this time), no other tools or commands can do this magic. Whenever you copy a sparse file to a new volume (disk), it is written out to disk in full. Whenever you copy two cloned files to a new volume, both files are written at full size, and the shared data is duplicated.
So, just the act of copying from the backup (even to an APFS disk) can expand these special files' disk requirements many-fold. In these cases, my understanding is that it doesn't matter if you copy with Finder, cp, rsync, or whatever. And that Apple has not released documentation or APIs to enable tools other than Time Machine to do this magic.
CCC tells me it doesn't support time machine backup file format and recommends to use finder.
Finder takes what should be a less than a terabyte file and says its 10+ terabyte, and after leaving for three days, only copied a quarter of it. It sure does seem like theres duplicative copying going on.
I'll give some of these other suggestions in this thread a try.
Finder takes what should be a less than a terabyte file and says its 10+ terabyte, and after leaving for three days, only copied a quarter of it. It sure does seem like theres duplicative copying going on.
I'll give some of these other suggestions in this thread a try.
That definitely seems to be the case. I recently added a new SSD to do TM Backups. Since I'm up-to-date on Mac OS, it formatted the drive to APFS. Finder can display those backups and can even drill down into them all the way to individual files. But a standard shell can't parse the backups. I can see directories in Finder, but the shell just says "Operation not permitted" when trying to get into them. That means existing tools other than Finder probably won't be able to manipulate the contents of the TM backups.
OP, if you want to do a full disk copy have you tried the "Restore" functionality in Disk Utility? I think that may be your only hope if you want to copy the contents of an APFS Volume with TM Backups to another drive.
If you are getting "Operation not permitted", try giving Terminal Full Disk Access in Security & Privacy -> Privacy
If you copy a single dated directory (snapshot) that will work.
Each snapshot is basically a full copy of all files. However on the Time Machine disk, hard links are used to keep multiple copies of the same file without using up disk space.
When you copy files to another drive, the hard links are translated into individual files and drive space is taken.
So copying multiple snapshots is complicated. rsync might help you. Or Disk Utility and CCC should allow you to image the Time Machine drive onto another disk.
Since I posted about this above, I did a little more reading. For awhile, I had CCC automatically clone the time machine volume on the 2012 Mini that I use as a file server and time machine destination. Seemed to work fine but I never tried to actually access the clone. Later, I re-purposed that drive and stopped cloning the time machine disk to it. Anyway, looks like I was wrong. See this:
Can I use CCC to copy a Time Machine backup?
No. Copying a Time Machine backup volume with anything other than the Finder is not supported (by us, nor Apple); CCC specifically disallows copying anything to or from a Time Machine backup volume. Apple does not document a procedure for making copies of Time Machine volumes.
Can I use CCC to restore content from a Time Machine backup?
Generally, no. If you want to restore content from a Time Machine backup, you should use Apple's Time Machine interface for that purpose. If you see a Time Machine snapshot in CCC's Snapshots table, however, you may restore files from that snapshot.
However, here's an older support article from Bombich that may shed some light on this, since I was using a shared network disk:
Backing up Time Machine sparsebundle disk images
When Time Machine is configured to back up a Macintosh to a network volume (such as a Time Capsule device), Time Machine stores the backup in a sparsebundle disk image. CCC can copy these sparsebundle disk image files without any special configuration; simply choose your network volume as the source of your CCC backup task. In fact, CCC quite capably copies only the bands within the sparsebundle that have changed, so you can add CCC to this type of setup for a second tier backup to an offsite network share.