Big Sur finally uses APFS snapshots for Time Machine backups!

[zorinlynx]

I've been waiting for this feature a long time and I'm so happy to finally see it! Using APFS snapshots for backups is so much faster and more efficient than the old way of walking the entire filesystem and using hard links. I did some tests backing up my test installation and it just flies.

I'm a UNIX sysadmin and work extensively with ZFS and using ZFS snapshots for backups, and this is basically the same thing. I think this is one of the reasons Apple considered using ZFS so many years ago, but then decided to implement their own. Backups are going to be MUCH faster, especially deltas with lots of small files. No more TM sitting there for 20 minutes at "Backing up 646MB of 650MB" with the disks cranking away.

 

[Stephen.R]

So presumably it’s converted your TM drive to APFS (as opposed to forcing it to be hfs+ previously)?

 

[hevaKmaI]

Does this work if you back up to a NAS?

 

[zorinlynx]

So presumably it’s converted your TM drive to APFS (as opposed to forcing it to be hfs+ previously)?

I haven't tested that. I'm using a completely separate test machine and am backing up to an APFS disk.

I should try a backup to an HFS+ disk to see if it still backs up the old way for that.

 

[casperes1996]

I haven't tested that. I'm using a completely separate test machine and am backing up to an APFS disk.

I should try a backup to an HFS+ disk to see if it still backs up the old way for that.

I have Big Sur but am not going to use Time Machine with it, but I’m curious what your findings will be. If it will convert the drive or just continue the old way

 

[bdart2]

With the news that Big Sur will support Time Machine in APFS - I had the following issue arise today.

I just bought a new MBP a few weeks ago and today bought a WD easystore 8 TB HDD for use as both a time machine and to store pictures and videos - one partition for TM and the other for other files.

I was just about to reformat and started looking at the format types and saw APFS as the preferred format for newer OSs. Then I saw that Time Machine is not compatible with APFS, but would be in Big Sur.

Question which one should I do:

1.) Format the new HDD in MacOS Journaled and just partition part of the drive for Time Machine and the other for files?

or


2.) Format an old spare drive to MacOS journaled and use it as a TM backup now and now Format the new HDD in APFS and partition the drive as above and wait to move the TM backups from the old drive to the new drive once I install Big Sur later this year?

thanks in advance

 

[Taz Mangus]

With the news that Big Sur will support Time Machine in APFS - I had the following issue arise today.

I just bought a new MBP a few weeks ago and today bought a WD easystore 8 TB HDD for use as both a time machine and to store pictures and videos - one partition for TM and the other for other files.

I was just about to reformat and started looking at the format types and saw APFS as the preferred format for newer OSs. Then I saw that Time Machine is not compatible with APFS, but would be in Big Sur.

Question which one should I do:

1.) Format the new HDD in MacOS Journaled and just partition part of the drive for Time Machine and the other for files?

or


2.) Format an old spare drive to MacOS journaled and use it as a TM backup now and now Format the new HDD in APFS and partition the drive as above and wait to move the TM backups from the old drive to the new drive once I install Big Sur later this year?

thanks in advance

I am not sure I would use the same drive for backups and storing other valuable items. If the drive fails, for whatever reason, you have lost everything.

 

[JimmyjamesEU]

I've been waiting for this feature a long time and I'm so happy to finally see it! Using APFS snapshots for backups is so much faster and more efficient than the old way of walking the entire filesystem and using hard links. I did some tests backing up my test installation and it just flies.

I'm a UNIX sysadmin and work extensively with ZFS and using ZFS snapshots for backups, and this is basically the same thing. I think this is one of the reasons Apple considered using ZFS so many years ago, but then decided to implement their own. Backups are going to be MUCH faster, especially deltas with lots of small files. No more TM sitting there for 20 minutes at "Backing up 646MB of 650MB" with the disks cranking away.

Hmmm, not so much. It's true that apfs disks can now be used for Time Machine, and it's also true that snapshots are created on both the source and target volumes...

However, TM still isn't using a block based backup I'm afraid. It's easy enough to verify this. Do an initial backup (which is pretty fast), then change a large file just a little (for example, I created a 10 gig file before the first backup, then added 10 megs to it). Do a subsequent backup and you will see that the entire 10 gig + 10 meg file is recopied. Not blocks unfortunately, just file copies.

It's still an early beta, so perhaps that will come later. It is also true to say that they have improved performance of TM overall. Still, disappointing that block based backups still aren't here.

 

[zorinlynx]

Hmmm, not so much. It's true that apfs disks can now be used for Time Machine, and it's also true that snapshots are created on both the source and target volumes...

However, TM still isn't using a block based backup I'm afraid. It's easy enough to verify this. Do an initial backup (which is pretty fast), then change a large file just a little (for example, I created a 10 gig file before the first backup, then added 10 megs to it). Do a subsequent backup and you will see that the entire 10 gig + 10 meg file is recopied. Not blocks unfortunately, just file copies.

It's still an early beta, so perhaps that will come later. It is also true to say that they have improved performance of TM overall. Still, disappointing that block based backups still aren't here.

Did you verify that the entire 10GB file is being copied though? Maybe TM just tells you 10GB basing that information on what's changed?

Now I feel like testing this myself but unfortunately my test Macbook is at the office. Argh.

 

[JimmyjamesEU]

Did you verify that the entire 10GB file is being copied though? Maybe TM just tells you 10GB basing that information on what's changed?

Now I feel like testing this myself but unfortunately my test Macbook is at the office. Argh.

I got the disk size after the first backup and then after the subsequent backups. Each time the space had been consumed the total size of the file, as indicated by the Finder, disk utility, and via command line. I also used the 'asr' command line tool that does use block based transfer. The difference was noticeable. Block based initial backup was 4 minutes for a 30 gig disk. Subsequent delta transfers were around 10-20 seconds. Time machine was fast, but not nearly as fast as the block based method.

 

[chama98]

so, tell me.. Did anyone have their time machine drive converted automatically to APFS from HFS+? I backup to a NAS drive, so wondering if I need to migrate backups to a external thunderbolt drive or something.

 

[yangm]

so, tell me.. Did anyone have their time machine drive converted automatically to APFS from HFS+? I backup to a NAS drive, so wondering if I need to migrate backups to a external thunderbolt drive or something.

Mine is on a NAS (SMB3, zfs backed), it didn't (nor did it offer to) get converted into APFS.

Does this work if you back up to a NAS?

I've created another share, so it creates another backup, on my NAS (same as above) and it does default to a APFS sparsebundle.

 

[donawalt]

So it sounds like you will have to start with a freshly formatted disk to use Time Machine with APFS...is that correct?

 

[yangm]

Seems like it, at least for now. Though that's coming from the very first developer beta... There's still a possibility that the final release is going to introduce some upgrade path.

 

[zorinlynx]

I got the disk size after the first backup and then after the subsequent backups. Each time the space had been consumed the total size of the file, as indicated by the Finder, disk utility, and via command line. I also used the 'asr' command line tool that does use block based transfer. The difference was noticeable. Block based initial backup was 4 minutes for a 30 gig disk. Subsequent delta transfers were around 10-20 seconds. Time machine was fast, but not nearly as fast as the block based method.

Ugh, this is frustrating! Why would they half-ass the concept? My only theory is maybe they don't full trust the asr incremental stream code yet? I know Apple has to be super conservative with software they deploy to millions of users, but it's already been multiple OS releases so I'm surprised they wouldn't have the kinks worked out.

Either way, at least using snapshots is most likely more reliable than the crazy chains of hard links they were using before. One step at a time I suppose.

 

[mikecwest]

I hope that using APFS will make it faster on external SSD backup drives. TM is so dang slow.

 

[JimmyjamesEU]

Ugh, this is frustrating! Why would they half-ass the concept? My only theory is maybe they don't full trust the asr incremental stream code yet? I know Apple has to be super conservative with software they deploy to millions of users, but it's already been multiple OS releases so I'm surprised they wouldn't have the kinks worked out.

Either way, at least using snapshots is most likely more reliable than the crazy chains of hard links they were using before. One step at a time I suppose.

Yeah, I get your frustration believe me! I can only assume it's a work in progress. They haven't talked about it at WWDC or on the website at all, so they know it's not enough to boast about. The current method is probably the most flexible because it works with both hfs and apfs.

Also worth considering, currently macOS allows only 24 hourly snapshots. For delta snapshots to work, the previous snapshot must exist on both systems. If a user were to wait a week or so between backups, no deltas would be possible. The way other os's with cow filesystems deal with this is to have way more than 24 hourly snapshots. This could easily be done on macOS... but the problem here is mac users! I think apple is worried about users complaining that "all their space is being used up". I have seen hundreds of posts like this. Complaining, whinging about snapshots, where users on linux, freebsd etc know and appreciate the immense benefits they provide. Its a case where macOS is being held back by the ignorance of it's user base.

One good thing, is there are people like John Siracusa with knowledge and interest in apfs. He just asked about Time machine and delta backups on Twitter. Hopefully he can have some influence.

 

[casperes1996]

Yeah, I get your frustration believe me! I can only assume it's a work in progress. They haven't talked about it at WWDC or on the website at all, so they know it's not enough to boast about. The current method is probably the most flexible because it works with both hfs and apfs.

Also worth considering, currently macOS allows only 24 hourly snapshots. For delta snapshots to work, the previous snapshot must exist on both systems. If a user were to wait a week or so between backups, no deltas would be possible. The way other os's with cow filesystems deal with this is to have way more than 24 hourly snapshots. This could easily be done on macOS... but the problem here is mac users! I think apple is worried about users complaining that "all their space is being used up". I have seen hundreds of posts like this. Complaining, whinging about snapshots, where users on linux, freebsd etc know and appreciate the immense benefits they provide. Its a case where macOS is being held back by the ignorance of it's user base.

One good thing, is there are people like John Siracusa with knowledge and interest in apfs. He just asked about Time machine and delta backups on Twitter. Hopefully he can have some influence.

To be fair the issue with uninformed users is always the responsibility of the creator who failed to properly inform them.
There's no easy storage management section with a GUI showing snapshots and their storage usage and such. It's fine for us who just pop the Terminal, but instead of just saying that System is using over a hundred gigs, users would be more able to understand if the storage management section you can get to with About This Mac actually showed more information about Snapshots and allowed you to control it without needing to use diskutil ap snapshots /

 

[yangm]

I guess much of the frustration comes from: we've waited so long... for this? Given they're pretty much reusing the same machinery which powers the current HFS+ replication, why didn't they do this earlier? Or, since we've waited for so long, why isn't this a block based snapshot replication like asr has been doing?

 

[JimmyjamesEU]

To be fair the issue with uninformed users is always the responsibility of the creator who failed to properly inform them.
There's no easy storage management section with a GUI showing snapshots and their storage usage and such. It's fine for us who just pop the Terminal, but instead of just saying that System is using over a hundred gigs, users would be more able to understand if the storage management section you can get to with About This Mac actually showed more information about Snapshots and allowed you to control it without needing to use diskutil ap snapshots /

I definitely agree that the tools could and should be better. Space should be accurately reported and consistent. I'm not sure I agree that users have no responsibility to learn and educate themselves, but tbh I don't care enough to argue the point!

Some (hopefully) promising news. If you run 'strings' on the backupd utility deep in the bowels of Big Sur, you can see references to diffing snapshots and delta blocks, so it seems it's something they're working on at least.

 

[casperes1996]

I definitely agree that the tools could and should be better. Space should be accurately reported and consistent. I'm not sure I agree that users have no responsibility to learn and educate themselves, but tbh I don't care enough to argue the point!

Some (hopefully) promising news. If you run 'strings' on the backupd utility deep in the bowels of Big Sur, you can see references to diffing snapshots and delta blocks, so it seems it's something they're working on at least.

Users should educate themselves, yes. I agree with that too. But macOS is a platform used by a wide variety of people and you can't expect Joe Bluepants to investigate things that are completely irrelevant to the way he uses his computer without even any indication from the system that there is something to be investigated in the first place. It just won't let him save his new photo to the computer, saying it's out of space even though he hasn't saved that many things, and apparently system eats up a lot of space. I don't think it's unreasonable for Joe Bluepants to then just complain about macOS eating up all his disk space. There's nothing that tells him to look into snapshots and a fix for his issue. There's just "system eating space". And if you tell him to use Terminal, he'll rightfully look at you like you're suggesting the dark arts, because we, the software developers, have been making GUIs precisely for average Joe Bluepants, telling them not to be scared of computers. It clearly looks like it's a problem that the system is eating up his space to him, and not a helpful feature. It becomes the user's responsibility as soon as the information has been presented to them. It's OK to get pissy with someone you see every day if they never remember your name. It's not OK to be pissy with them the very first time when you've not even told them your name yet. It's the user's responsibility as soon as they've been given the tools to know it, but not before that. Before that it's our responsibility (our referring to developers and creators).
That's my opinion anyway

But yeah that does sound promising

 

[ErikGrim]

Convert to APFS is greyed out for my TimeCapsule, but the option to Erase it to APFS is there. Is it safe to do so?

 

[mikecwest]

Users should educate themselves, yes. I agree with that too. But macOS is a platform used by a wide variety of people and you can't expect Joe Bluepants to investigate things that are completely irrelevant to the way he uses his computer without even any indication from the system that there is something to be investigated in the first place. It just won't let him save his new photo to the computer, saying it's out of space even though he hasn't saved that many things, and apparently system eats up a lot of space. I don't think it's unreasonable for Joe Bluepants to then just complain about macOS eating up all his disk space. There's nothing that tells him to look into snapshots and a fix for his issue. There's just "system eating space". And if you tell him to use Terminal, he'll rightfully look at you like you're suggesting the dark arts, because we, the software developers, have been making GUIs precisely for average Joe Bluepants, telling them not to be scared of computers. It clearly looks like it's a problem that the system is eating up his space to him, and not a helpful feature. It becomes the user's responsibility as soon as the information has been presented to them. It's OK to get pissy with someone you see every day if they never remember your name. It's not OK to be pissy with them the very first time when you've not even told them your name yet. It's the user's responsibility as soon as they've been given the tools to know it, but not before that. Before that it's our responsibility (our referring to developers and creators).
That's my opinion anyway

But yeah that does sound promising

I agree, most end users would not ben know what any of that means. I like the old school Mac logic, that is "just works."

 

[gilby101]

They haven't talked about it at WWDC or on the website at all, so they know it's not enough to boast about.

My interpretation:
Apple would not want boast about a feature that might or might not be in the final release. My guess is that Apple are wanting feedback from beta testers before deciding to include it. Remember 2 years ago there was APFS on Fusion drives in the beta, but not in the release.

 

[zorinlynx]

Convert to APFS is greyed out for my TimeCapsule, but the option to Erase it to APFS is there. Is it safe to do so?

Make sure you have an additional backup of your system before wiping the current backup.

It would really suck if you wiped your current backup then had a storage failure while creating a new backup.

Of course if you're just on a test system this is moot.