Time Machine backups painfully slow on NAS (smb)

[mfram]

I've been lurking on this thread. I want to report that it is possible to run Time Machine to an SMB server with very nice performance if you have control over the whole chain. Admittedly in my case, I have pretty beefy hardware for the SMB server. It's a Core i7 PC running Linux with Samba 4.13.9. The disks with the TM mount are running on top of LVM on a 5-disk RAID 5 array with 7200 RPM disks. The underlying filesystem on the Linux side is standard ext4. My MBP is running with wired ethernet typically. Wireless seems to make it a little slower. Big Sur gives some goofy large estimated how long it will take to do a backup. But the backup never takes as long as it claims.

I have an old Time Machine access point I use as another backup server. But this one goes a lot faster.

I put together my Samba conf from information across various pages on the Internet and it seems to work great. The Samba server shows up when you add a new TM Network disk and works naturally. Here are important bits in my SMB configuration related to this functionality.

[global]
  max protocol = smb
  client max protocol = smb
  unix extensions = no
  wide links = yes
  create mask = 0644
  encrypt passwords = yes
  smb encrypt = auto
smb passwd file = /etc/samba/private/smbpasswd
  local master = yes
  domain master = yes

# Time Machine
ea support = yes
vfs objects = catia fruit streams_xattr
durable handles = yes
kernel oplocks = no
kernel share modes = no
posix locking = no
fruit:advertise_fullsync = true
smb2 leases = yes
map to guest = Bad User

[timemachine]
   comment = Time Machine
   path = /mnt/timemachine
   writeable = yes
   browseable = yes
   create mask = 0600
   directory mask = 0700
   spotlight = yes
   vfs objects = catia fruit streams_xattr
   fruit:aapl = yes
   fruit:metadata = stream
   fruit:model = MacSamba
   fruit:time machine = yes
   inherit acls = yes

Oh, and mdns is also involved. Here is part of the avahi configuration on the server. I think the important part is the _adisk._tcp service.

<?xml version="1.0" standalone='no'?>
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
 <name replace-wildcards="yes">%h</name>
  <service>
     <type>_adisk._tcp</type>
        <txt-record>sys=waMa=0,adVF=0x100</txt-record>
           <txt-record>dk0=adVN=timemachine,adVF=0x82</txt-record>
            </service>
              <service>
                  <type>_smb._tcp</type>
                      <port>445</port>
                        </service>
                        </service-group>

 

[southerndoc]

Reviving the thread here... This slow backup problem is still occurring with 11.5.

One thing I have a problem with is when TM is preparing for backup or backing up, I have periodic network lags that cause page timeouts with Google Chrome. This has occurred with my M1 MM built-in 10G networking, a SonnetTech SFP+ adapter, and with an older 2012 iMac. Oddly enough, I do not have this problem with my M1 MBA backing up over WiFi (at least that I've noticed).

 

[dim_]

A minor note, Big Sur 11.5 still fully reindexes all backups after each incremental backup. I.e. any slowness *after* successful backups is still not fixed.

 

[haralds]

Big Sur changed its backup filesystem and approach to using APFS. There is a dramatic difference in using snapshots instead of layers of hard links. It dramatically improves speed and reliability.
More about that in this article https://eclecticlight.co/2021/04/15/time-machine-to-apfs-first-full-backup/ and others on this site.
I confirmed that a new backup to a networked TimeMachine volume including Time Capsule now produces a sparse disk formatted in the APFS file system in the same way as local TMS disk backup is.
You might have to start with a fresh backup after upgrading to Big Sur.
The speed of the networked backup is still limited by network and SMB performance.

 

[southerndoc]

My issues occurred with new backups from new machines (Mac mini M1 and MBA M1). The MBA doesn't have the issue, but the Mm M1 does.

 

[dim_]

Big Sur changed its backup filesystem and approach to using APFS. There is a dramatic difference in using snapshots instead of layers of hard links. It dramatically improves speed and reliability.

Well, it doesn't help for the issue that it completely reindexes the backups every time, leading to severe slowness due to Spotlight. My backups are all APFS and made by Big Sur.

 

[foxfire_pt]

Well, it doesn't help for the issue that it completely reindexes the backups every time, leading to severe slowness due to Spotlight. My backups are all APFS and made by Big Sur.

Iup and you can not disable the reindexing on a timemachine drive. I have a Qnap and i gave up on that for now. My linux machine also used to work great, but now seams to have the same problem. i was using a gigabit hardwired network for backup, so that was not an issue. If i used wifi it got even worse, even completly unusable.
I solved it by using an external usb disk. works great. the only problem is sometimes i forget to connect it ;-), but if you connect it on your dock problem solved.

 

[ARMunix]

Well, it doesn't help for the issue that it completely reindexes the backups every time, leading to severe slowness due to Spotlight. My backups are all APFS and made by Big Sur.

I'm now pretty sure it does not reindex all of the data - would take much longer. But mds is clearly going over some of the files - maybe it's the ones can't index properly, so it's trying again after very backup run?

Of course it's taking longer as the number of backups increases. Just after the my last backup, th process took about 30 minutes...

 

[dim_]

Just installed the most recent Big Sur 11.5.2 update, which had a very short description "fix bugz "

Unfortunately it did not fix the reindexing issue. It's still happily chugging through everything after each incremental backup.

 

[southerndoc]

I just deleted my TM backups from my Synology NAS. I had a custom username for the TM backups. I deleted those. Went with my username that my Mm M1 and MBA M1 use to access my NAS home folder, but I granted access to the TM folder.

After deleting the TM backups from the NAS, I clicked on the disk TM is currently using and removed the disk. I then selected the folder where my TM backups are to go on my NAS, used the encrypted backup option, and started all over.

Since then, I'm not having any of the network lag problems like I was before. Fingers crossed that this has fixed it!

 

[southerndoc]

I was able to resolve my issue by turning off Time Machine broadcast for SMB in Synology DSM. I kept it enabled for AFP. It seems that Time Machine backups via AFP do not cause the network issues that I experienced with SMB.

 

[EugeneKey]

I was able to resolve my issue by turning off Time Machine broadcast for SMB in Synology DSM. I kept it enabled for AFP. It seems that Time Machine backups via AFP do not cause the network issues that I experienced with SMB.

I turned off TM for SMB and enable AFP on my Synology NAS
I'm using MacBook Air M1 with BigSur 11.6. It may have speed up the backup process a bit, but it didn’t solve the issue with infinite file reading. Still the same: kernel_task use the network with constant speed of 20Mbit after finished the backup process. In the logs, I see that the reading goes through the AFP protocol from different files inside sparsebundle.
Because of this, I had to abandon TM backup over the network. I use TM backup to an external drive connected locally and it works fine.

 

[petterihiisila]

I wanted to try something new.

"Rules" - I treated these as design targets:

1. No fans and no spinning disks, must be 0 dB
2. M1 Air normally docked, but can be undocked at any time, without a warning, during the day
3. Time Machine backup with hourly snapshots, with no manual mounting/unmounting
4. WiFi only for the M1 Air, so undocking doesn't disconnect/switch any network adapters
5. Increments must take 5 minutes or less (normally)
6. Reliable eject, when a manual eject is needed

Those are somewhat conflicting goals, but it worked out. I previously had a local HDD, but that violates rule #1. I had a local SSD, but that violates rule #2. I had an HDD behind Raspberry Pi via a LAN path, but that violated almost all the rules.

The current setup is this:

• Raspberry Pi 4B with 4 gigs of RAM, no cooling/fans, connected to a Wifi mesh box via Ethernet
• Samsung T5 SSD, 2 TB, connected to Pi via USB3, formatted as XFS (ext4 is probably a wiser choice)
• OpenMediaVault SMB/CIFS share with Time Machine support checked
• Macbook Air M1, connected via WiFi only (I did use wired LAN for the first big backup, that's not cheating)
• No other disks connected to the Mac, wired or otherwise

And here are the results, addressing points 1-6 above:

1. Indeed no fans or disks, 100% silent.
2. TimeMachineEditor is set to only back up between 02 and 07 AM. Time Machine disk gets auto-ejected before the workday. This means that nothing external is mounted during the day and undocking is safe without a warning.
3. TimeMachineEditor keeps scheduling hourly snapshots into the M1 SSD, which are then offloaded to the NAS overnight by Time Machine.
4. LAN was only required (or, convenient) during the initial backup. Now the M1 is WiFi only, and existing connections stay up when undocking.
5. The the configuration is plenty fast, a normal update takes 5 minutes or less over WiFi.
6. Whatever the Mac is doing while it doesn't want to eject, it does it much faster with Pi+SSD than Pi+HDD. I haven't had to use force or wait for long, so far.

The initial backup (wired) ran at about 300-500 Mbps, around 2-3 GB/minute. That's pretty good for a 0 dB low-cost NAS IMO. Poor Raspberry Pi 😍 had to throttle its CPU all the way to the finish line, but now that it's there, it's running cool (enough) for incremental updates.

I haven't had this configuration for very long. If things yet again go boom over time, I'll write an update.

 

[val74]

So far the only way I've found to be useful in order to stop the re-indexing after the TM backup occurred on my NAS is to manually unmount the TM disk image from Disk Utility. Anyway this thing has been going on since Big Sur's update 11.3 and has not been addressed in any subsequent update nor in Monterey. You can check in Activity Monitor that after the TM update has finished, some mdworker_shared instances are spawn, and they read in the TM backup from its initial date. This is very annoying, especially when on wifi and on battery (it also affects battery life), and I can't really understand the point in that.

 

[RiderX]

My TM backups are triggered by a daily cron job (tmutil …) and after it has finished I do a „diskutil unmount“. Works well and no more hours long indexing sessions.

 

[val74]

My TM backups are triggered by a daily cron job (tmutil …) and after it has finished I do a „diskutil unmount“. Works well and no more hours long indexing sessions.

I have never had any slowdown problem with my TM over SMB solution.
Everything worked flawlessly until Big Sur 11.3 update, when APFS was chosen over HFS+ for the TM sparsebundle images too.

This is something they should address, rather than we users have to deal with creating any sort of weird solutions and trickery (with all the due respect for your idea).

Obviously there is a bug with Spotlight not being able to understand that the TM Disk Image doesn't need to be re-indexed in its entirety from backup day 1, but just for the latest backup instead.

Frankly this operation should also be done as part of the TM backup session; in this way the user could eventually cancel it altogether with the TM backup if they need to put to sleep the machine at that moment and cannot wait for the whole operation to complete.

Indeed we don't even know if unmounting the TM disk image while those re-indexing are taking place is a safe operation that won't compromise anything in the TM backup Spotlight indexing.

One more thing in regards to the UI: the TM percentage indicator (now on Monterey, before on Big Sur was the data size of the backup) runs till ~70% of the backup, then the whole operation concludes and goes into "cleanup" state, leaving the user wondering if everything has been really backed-up or if the backup got interrupted by itself for some reason.

Sorry about the long rant, I don't usually do that, but this thing is really annoying and also hard to troubleshoot, which frankly is becoming a —bad— trend in macOS lately.

 

[val74]

I turned off TM for SMB and enable AFP on my Synology NAS
I'm using MacBook Air M1 with BigSur 11.6. It may have speed up the backup process a bit, but it didn’t solve the issue with infinite file reading. Still the same: kernel_task use the network with constant speed of 20Mbit after finished the backup process. In the logs, I see that the reading goes through the AFP protocol from different files inside sparsebundle.
Because of this, I had to abandon TM backup over the network. I use TM backup to an external drive connected locally and it works fine.

I've opted for not letting the TM backups go on while on battery power.

This way I rely on local snapshots which then get pushed to the NAS when I plug into power my M1 MBPro (usually overnight).

Not an ideal solution in case —finger crossed— the internal SSD gets corrupted: I'll have available just the TM backup from one or two days ago.

Hence I also manually start a TM backup from time to time when on battery power too. But it's really annoying having to wait that the Spotlight index happen for the whole TM backup disk image as the TM Backup ends, especially now that such backup is 6 months old.

 

[val74]

I'm now pretty sure it does not reindex all of the data - would take much longer. But mds is clearly going over some of the files - maybe it's the ones can't index properly, so it's trying again after very backup run?

Of course it's taking longer as the number of backups increases. Just after the my last backup, th process took about 30 minutes...

I think it does.

I've been looking in the mdworker_shared processes' "files and ports open" via Activity Monitor, and backups from day 1 are re-indexed every time.

Moreover this happens for every "concrete" user on the system (I keep an administrator user with administrator privileges and an user with general purpose user privileges on my system) and for _spotlight user too.

 

[rehkram]

I think it does.

I've been looking in the mdworker_shared processes' "files and ports open" via Activity Monitor, and backups from day 1 are re-indexed every time.

Moreover this happens for every "concrete" user on the system (I keep an administrator user with administrator privileges and an user with general purpose user privileges on my system) and for _spotlight user too.

That's a very interesting finding. Kinda makes me wonder if the NAS system s/w is doing, er, "something" to the physical files that triggers mdworker to get a false positive during a change detection routine. Just a crazy wild guess.

Having said that, my Syno NAS 1511+ TM backup seems to run trouble free at the moment. I have both SMB & AFP enabled on the NAS. Due to my low usage and desire to reduce risk of conflicts, I only usually run backups once a week, using the TimeMachineEditor app. I trigger a backup manually if I need more than that, e.g. before an upgrade.

 

[val74]

That's a very interesting finding. Kinda makes me wonder if the NAS system s/w is doing, er, "something" to the physical files that triggers mdworker to get a false positive during a change detection routine. Just a crazy wild guess.

Having said that, my Syno NAS 1511+ TM backup seems to run trouble free at the moment. I have both SMB & AFP enabled on the NAS. Due to my low usage and desire to reduce risk of conflicts, I only usually run backups once a week, using the TimeMachineEditor app. I trigger a backup manually if I need more than that, e.g. before an upgrade.

I don't think so. TM Backups are encrypted and stored into a sparsbundle disk image, the NAS filesystem shouldn't do anything at all with it.

Besides this problem has arisen since TM backups over network disk images had been moved to APFS filesystem rather HFS+, hence since Big Sur 11.3 update. To me it looks like more as there is a problem with the new architecture of TM backups on APFS not taking into account when the backup is towards a disk image rather than a physical drive.

I'm using a Synology NAS too, and as I stated in another comment I never had any problem with TM backups over it via SMB until this problem of reindexing arisen in Big Sur 11.3: no corruptions, no slowdowns, backups on battery too and scheduled every hour.

 

[ARMunix]

I’ve given up on TM backups to my NAS - I got a cheap external drive and use that instead. Backups are way faster. Even backing up to an old external USB 2.0 drive seemed faster.

One thing I noticed with the external drive: re-indexing does happen too, but when the backup drive is mounted, not after every backup.

 

[val74]

I’ve given up on TM backups to my NAS - I got a cheap external drive and use that instead. Backups are way faster. Even backing up to an old external USB 2.0 drive seemed faster.

One thing I noticed with the external drive: re-indexing does happen too, but when the backup drive is mounted, not after every backup.

I have to check it out on my iMac which is set to to TM Backups on an external physical drive. If I remember well I had the same behaviour also on a physical drive when I first investigated this issue, which anyway wasn't impacting that much as it does for backups done over the network and on a laptop that runs on battery power…

 

[rg_1980]

Hello,

first post here, I have been lurking these forums for some months, time to contribute with something (hopefully) useful.

I have been suffering this since late Big Sur versions over M1 Macbook Air and then on Monterey with a 2021 16 MBPro.
Had some time to work on my Synology NAS and think I solved the issue (at least, seems solved for me).

Not sure this has been posted already but please check this:

Control Panel -> File services -> Under AFP/SMB -> Advanced Settings -> Apply predefined Unix Authentications -> Active

(menus/section may be a little different but I think you got the point, non-english DSM here)

Started a fresh backup in this moment and I am transferring at about 20-30 MBps (not Mbps) via AFP/WiFi-5.

Please note I'm on the latest DSM V6, not yet switched to V7.

Hope this helps.

 

[rehkram]

On mine (English) it's labeled "Apply default UNIX permissions".

I'm not having any problems with slowness. Having said that, I've enabled that option in DSM and will see what happens after the backup I've just started.

 

[rehkram]

That ran very fast but I've been booted into macOS on the other drive for a few days so few- to no changes since the last backup. First impression though is 'wow, that was fast'.

edit later: I went into TM and restored a file from the backup I just did, worked fine. I'll play around with this for a while and provide feedback from testing it. I'm also on DSM V6, my Syno NAS is an oldie, 1511+. But I don't think that matters in this case; it's always been about permissions, IMO.