Big Sur finally uses APFS snapshots for Time Machine backups!

[Marx55]

When you open and close a virtual OS of Mac, Windows or Linux-Ubuntu with applications like VMware Fusion in Mac, Time Machine of macOS before 11 Big Sur requires HFS+ and makes a full copy of the virtual OS disk image, which is usually huge, even when nothing has been changed inside the virtual machine except perhaps the dates os the files or a little more.

My question is: does the new Time Machine of Big Sur on APFS disks make such virtual OS disk image backup much faster, copying only the changed files (if any) inside the virtual disk image, or does it behave as previous Time Machine versions on HFS+ disks, copying the whole virtual OS disk image and filling the Time Machine disk quickly?

 

[gilby101]

My question is: does the new Time Machine of Big Sur on APFS disks make such virtual OS disk image backup much faster, copying only the changed files (if any) inside the virtual disk image, or does it behave as previous Time Machine versions on HFS+ disks, copying the whole virtual OS disk image and filling the Time Machine disk quickly?

You need to experiment with that and report back as I am sure others would be interested in the results. As far as I can tell there is no consensus on the performance and storage of VMs. And certainly no authoritative conclusion. Also, Apple, has not (surprise, surprise!) documented APFS TM to the detail we would like.

There is some discussion of VM to TM(APFS) in the comments to Getting started with Time Machine to APFS in Big Sur.

My understanding is that TM's use of APFS does not resolve the question of whether TM backups of VMs are (or even can be) fully reliable. At best, the backup would be as if the plug was pulled out on the VM - potentially worse. The improvements in reliability came longer ago when TM started using APFS snapshots of the source disk.

Have a read of What is wrong with Time Machine? I have decided to use his Vimalin utility which can be scheduled to take regular VM snapshots using VMware's offical method of taking snapshots. [This is why I have not experimented with performance of TM backing up VMs.]

 

[Marx55]

You need to experiment with that and report back as I am sure others would be interested in the results.

Thanks. I asked because I am not using macOS 11 Big Sur (having Time Machine with such feature of saving backups into APFS disks), and I wonder how it behaves with virtual disk images. If someone tries, please post here. Thanks!

 

[hoodafoo]

I don't back up my VM's because I think it's too disk space intensive, plus I have non-provisioned disks. Every session, gigs of data get changed. If only there was a way to backup only actually files of the VM and not memory states

 

[chrfr]

If the VM is shut down at the time Time Machine runs, there's no issue restoring from a backup. I've done it in the past without a problem. Restoring a running VM is likely to lead to unexpected results.

 

[ItWasNotMe]

I don't back up my VM's because I think it's too disk space intensive, plus I have non-provisioned disks. Every session, gigs of data get changed. If only there was a way to backup only actually files of the VM and not memory states

What I do is (using Parallels)

• Exclude the VM's from Time Machine
• For each VM, have a folder structure outside the VM that holds the data files for that VM. These are both visible to the VM and backed up using Time Machine plus my other backup software
• Have a folder visible to Time Machine into which I periodically copy the VM whilst it's not active. This is done after each major upgrade of the VM

All that sits inside the VM is the OS and any applications I've installed on that OS

That way, I don't write gigs of data, just the changes to the data that the VM has created and I have a backup to a recent state of the VM, conceding that this may be slightly out-of-date.

 

[gilby101]

If the VM is shut down at the time Time Machine runs, there's no issue restoring from a backup. I've done it in the past without a problem. Restoring a running VM is likely to lead to unexpected results.

I agree you can safely restore from a TM backup taken when a VM is shutdown. But TM does backups every hour and sometimes those will be of the shutdown VM and sometimes of it running. Do you have a record of which? And what happens when TM thins backups to once a day - it may well be retaining backups of running VMs and removing the shutdown ones.

Much safer to use another strategy. For examples, backup files from within the TM, use VM snapshots or just occasional copies of shutdown VMs - whatever suits your recovery need.

 

[chrfr]

I agree you can safely restore from a TM backup taken when a VM is shutdown. But TM does backups every hour and sometimes those will be of the shutdown VM and sometimes of it running. Do you have a record of which? And what happens when TM thins backups to once a day - it may well be retaining backups of running VMs and removing the shutdown ones.

Much safer to use another strategy. For examples, backup files from within the TM, use VM snapshots or just occasional copies of shutdown VMs - whatever suits your recovery need.

I can't disagree. Most of my VMs are essentially throwaways for software deployment testing so I don't worry about those. For the one that's not, it's not practical to store documents outside the VM, so I rely on Fusion's autoprotect snapshots and periodically manually make an archival copy.

 

[zorinlynx]

I agree you can safely restore from a TM backup taken when a VM is shutdown. But TM does backups every hour and sometimes those will be of the shutdown VM and sometimes of it running. Do you have a record of which? And what happens when TM thins backups to once a day - it may well be retaining backups of running VMs and removing the shutdown ones.

Much safer to use another strategy. For examples, backup files from within the TM, use VM snapshots or just occasional copies of shutdown VMs - whatever suits your recovery need.

Before TM makes an hourly backup, it creates a snapshot of the filesystem the VM is on. This snapshot is instant and atomic, a moment in time. The VM disk image is not being copied while it is being written to.

This means APFS-based backups of a VM image will be a lot more reliable than before when HFS was used. There's still always the risk that there will be some corruption if the VM is running, but the risk is a lot lower, and since you have hourly backups, if the most recent backup has a corrupted disk image you can always go back to the one before.

One trick you can use to restore a consistent backup of a running VM is to try to find an hourly backup where the timestamp on the VM disk image is a minute or more before the time of the backup. This means the image was relatively quiescent when the snapshot was made so it should be consistent.

I think operating systems in general need a reliable way to back up VMs. Instant snapshots get you a lot of the way there, but it would be nice if a backup process could tell all VMs running on a host to "sync", then freeze writes until the snapshot is made. We have similar concerns backing up filesystems with SQL databases, but there are easy ways around it, like using a journal file and having the database server dump a backup to a scratch disk periodically.

 

[ducknalddon]

Before TM makes an hourly backup, it creates a snapshot of the filesystem the VM is on. This snapshot is instant and atomic, a moment in time. The VM disk image is not being copied while it is being written to.

It's only atomic from the perspective of the file system. If Parallels/VMWare decides to write two files in succession and the snapshot starts between those writes then only one of them will be in the snapshot even though they might need to be paired.

 

[zorinlynx]

It's only atomic from the perspective of the file system. If Parallels/VMWare decides to write two files in succession and the snapshot starts between those writes then only one of them will be in the snapshot even though they might need to be paired.

This is true, which is why I said there's still risk of corruption.

However, the risk is MUCH lower than it used to be back when Time Machine would copy the VM file, multiple gigs from beginning to end, with the VM able to write to the file being copied during that entire time.

 

[BigMcGuire]

I just bought a LaCie 4TB drive to use as my TM drive and finally got an AFPS TimeMachine. WOW this is amazing. So cool! I love how I can create a partition that shares the space with the TM partition. Just amazing.

I manually copied my Parallels instance to the TM drive (am doing this right now). <shrug> just what I've always done with Parallels.

Either this LaCie drive is insanely fast or it backed up my 300GB MacBook in the fastest time I've ever experienced.... I'm pretty happy with this.

 

[BigMcGuire]

I did the same as you and mine also now shows case-sensitive. I am positive I selected only APFS Encrypted when I formatted, so TM must have switched it to case-sensitive.

I am actually noticing pretty good speed increase with TM compared to HFS+.

Same thing happened to me. I intentionally did APFS Encrypted and it switched it to APFS case-sensitive encrypted. Also noticing a massive difference in speed - the first backup especially!

 

[BigMcGuire]

in disk utility.

Time Machine partition ^

And a backup partition I can copy/paste data to (like my Parallels partition).

And in finder:

and what it looks like when doing a backup (in finder):

The first volume won't let me store anything in it but the second volume shares the space on the disk with the Time Machine partition.


What the Time Machine partition looks like in finder:

Cool!

 

[tuckerdogavl]

A few questions:

I had a problem upgrading to Big Sur last month, but I got my TM backup into the Big Sur OS (I was using Mojave). I still have that backup on a TM designated HD.

I had another External HD I had used on a MBP that I didn't need TM to backup any more. It is partitioned (1.5gb; .5gb) The 1.5gb I designated for the TM.

Does Big Sur not count "down" the TM backup now? I started the backup and it said "10gb backing up" and then went to "cleaning up" after about 10 minutes.

And, it is still "cleaning up" (for about ten minutes and counting). But I didn't see anything indicating anything happened. Yet. Is this normal?

I also discovered that when I wiped the 1.5gb of the 2TB on the External that is partitioned, it automatically created the APFS. (see below). So, I assume that was a good thing?