DaisyDisk very slow to scan in Sonoma

[Morac]

When I run DaisyDisk (web or App Store version) in Sonoma, it takes a very long time to scan. At least 5 minutes. In Ventura it used to scan in under 30 seconds. Sonoma has full disk access. I see syspolicyd, tccd and trustd using a lot of cpu while DaisyDisk is scanning.

Is anyone else seeing this problem?

 

[Mac Hammer Fan]

Perhaps DaisyDisk needs an update.
I only use Sonoma on an external SSD at this moment. I am waiting for MacOS 14.1

 

[gilby101]

Is anyone else seeing this problem?

DaisyDisk (4.25) is as fast as ever - maybe faster.

 

[Mike Boreham]

When I run DaisyDisk (web or App Store version) in Sonoma, it takes a very long time to scan. At least 5 minutes. In Ventura it used to scan in under 30 seconds. Sonoma has full disk access. I see syspolicyd, tccd and trustd using a lot of cpu while DaisyDisk is scanning.

Is anyone else seeing this problem?

Yes slow for me also.

 

[Oleg K.]

The developer of DaisyDisk here. I have received a few reports regarding slow or stalled scanning on macOS Sonoma, but it seems that only a very small number of users are affected. I'm actively investigating the cause of the issue, but unfortunately it doesn't reproduce on none of my systems. So please send all such reports to support@daisydiskapp.com.

In general, the current version of DaisyDisk should work fine on macOS Sonoma, because no breaking changes were introduced with it. But of course, another DaisyDisk update is also coming - hopefully with this issue addressed as well. Thanks!

 

[C0ppert0p]

I'm not seeing a slowdown when I scan my 2TB SSD. External disks are a problem though. I have two 5TB Passport drives, that take so long to do, its not worth the effort. (it would be awesome if scans could be kept, because by external drives are for archive and backup use only. I'd never scan them more than twice a year)

 

[gilby101]

I have two 5TB Passport drives, that take so long to do, its not worth the effort.

I suspect it is a sign of fragmentation. My 3TB Passport drive scans very fast, but was recently reformatted. My slightly faster 6TB MyBook is slow (it does complete in a usable time) - it is nearly 3 years since reformat and I am sure it is highly fragmented.

Edit: Are your backups Time Machine (or CCC, etc) with snapshots? Lots of snapshots slows the scan.

 

[Morac]

I’ve determine that directories that have a lot of files in them can scan really slowly. I deleted up a folder with 22,000 files in it and that cut about a minute off the scan. Never had to do that before though. Also sometimes it just is very slow and other times just kind of slow.

 

[Oleg K.]

The DaisyDisk developer here. Turns out the slowdown on macOS Sonoma happens in the folders ~/Library/Containers and ~/Library/Group Containers and has something to do with the new level of access control that Apple has added in macOS Sonoma when accessing data of other sandboxed apps. I'm currently working on a workaround.

> External disks are a problem though.

I don't think this is related to macOS Sonoma. The scanning speed depends on many factors - type of the media, its file system and the number of files (not their total space). If you are using the external disk for backing up your disk on a regular basis, i.e. your external disk contains multiple copies of your startup disk, it's easily possible that the disk contains extraordinarily large number of files. Besides, the huge number of files causes high memory usage to store the report. That perhaps explains the slowdown. The work around is to use the Scan Folder button and scan by folder separately.

 

[Morac]

The DaisyDisk developer here. Turns out the slowdown on macOS Sonoma happens in the folders ~/Library/Containers and ~/Library/Group Containers and has something to do with the new level of access control that Apple has added in macOS Sonoma when accessing data of other sandboxed apps. I'm currently working on a workaround.

That would explain the App Store version, but why would the web version be sandboxed?

In any case I tested the newer version with the workaround and it didn’t seem to make a difference.

 

[auxbuss]

The DaisyDisk developer here. Turns out the slowdown on macOS Sonoma happens in the folders ~/Library/Containers and ~/Library/Group Containers and has something to do with the new level of access control that Apple has added in macOS Sonoma when accessing data of other sandboxed apps.

fyi, I use the command line app ncdu (NCurses Disk Usage) and experience the same issue with ~/Library/Containers, but not ~/Library/Group Containers. The slowdown is from about 5 secs on Monterey to 20 secs on Sonoma.

 

[krbrock1]

definitely a slow down after updating to Sonoma from Ventura. Ventura would scan my MBP 500GB internal SSD in less than 1 min - I just ran it for the first time after installing Sonoma and the latest update 14.1 - the scan took a little over 3 min and seems to really hang around 20% for the bulk of that time.

 

[Oleg K.]

Guys, I'd like to update you with my findings.

The slowdown of DaisyDisk scanning on Sonoma affects only the following folders:

~/Library/Containers
~/Library/Group Containers

These folders conventionally contain data created by all sandboxed apps.

The slowdown is caused by a new privacy feature in Sonoma that additionally limits access to the app's data - you may see a new popup dialog asking your permission to scan apps' data when you scan your disk in DaisyDisk.

It's unclear however why, after granting such permission or after granting the "full disk access" to DaisyDisk, macOS Sonoma still performs some very expensive access checks for each subfolder of the mentioned folders, which results in the noticeable slowdown of scanning. At average, subfolders of this folder take 10-100x longer to open compared to regular folders. Luckily, not many users have a lot of subfolders there, so the number of affected users will likely be small.

I've contacted Apple engineers via the Developers Support and they have concluded it was a bug of macOS Sonoma and requested me to report it as a bug, which I have done. Hopefully, Apple will fix it in future updates of macOS Sonoma, but it's hard to get a time estimation for this.

In the meantime I'm working on a temporary workaround in DaisyDisk. I will post an update once it's ready.

 

[trickyslip]

Thank you for the update and the awesome app!

 

[krbrock1]

Thanks for the update.

 

[TheTerminalMan]

This isn't a problem specific to DaisyDisk, as the same kind of slowdown can be demonstrated even using basic commands from the shell. It appears that directory accesses (file lookups, file creations) within the Data subdirectories within the ~/Library/Containers tree are drastically slowed down.

I wrote a shell script that creates a bunch of directories (100 demonstrates the problem pretty well) within a Data directory of an existing app tree and pretty much anything I run (find, ls, rmdir, etc) that accesses the directory tree demonstrates the slowness.

Maybe it's because DaisyDisk and commands running from shell/terminal aren't the ones for which the Containers trees were created? Or... could it be that even the "owners" of the Containers trees are also suffering if they're doing heavy directory accesses?

 

[Oleg K.]

You are totally right, this problem is not specific to DaisyDisk. The ~/Library/Containers folder is special because it conventionally contains app data of all sandboxed apps. In Sonoma, Apple has added some additional privacy checks when apps are attempting to access this folder and its subfolders - you may see a new security popup during scanning. The checks are implemented on the core level of macOS, so all apps will be affected. Unfortunately, the implementation of this feature was obviously suboptimal and adds significant delays when accessing this folder.

If you send me your script to support@daisydiskapp.com, I can append it to my bug radar report, as another proof of the problem. Thanks!

 

[Czo]

You are totally right, this problem is not specific to DaisyDisk. The ~/Library/Containers folder is special because it conventionally contains app data of all sandboxed apps. In Sonoma, Apple has added some additional privacy checks when apps are attempting to access this folder and its subfolders - you may see a new security popup during scanning. The checks are implemented on the core level of macOS, so all apps will be affected. Unfortunately, the implementation of this feature was obviously suboptimal and adds significant delays when accessing this folder.

If you send me your script to support@daisydiskapp.com, I can append it to my bug radar report, as another proof of the problem. Thanks!

Meantime, can you add an option to the settings to ignore those folders (or a user editable list, where anyone can add folders)? It's will be (for me especially) a good workaround until Apple fix this problem. Because, currently DaisyDisk is unusable.

 

[Oleg K.]

Meantime, can you add an option to the settings to ignore those folders (or a user editable list, where anyone can add folders)? It's will be (for me especially) a good workaround until Apple fix this problem. Because, currently DaisyDisk is unusable.

I thought about it, but didn't appeal to me as a good solution, for the following reasons:

1. Not all Sonoma users experience this problem, it only becomes visible perhaps when there are lots of subfolders in ~/Library/Containers. It's impossible to tell in advance whether the scan will stall or not.
2. Even those who experience the problem report that rebooting the Mac temporarily fixes it. (BTW, have you tried this?). Therefore the criterion for applying the workaround becomes more blurry.
3. The ~/Library/Containers includes app data, which is often the source of disk space consumption. Excluding it from the report will potentially add a big chunk of "hidden space", and make it unusable in another way. It will be hard to explain it to the user why he sees the hidden space, and it's easy to forget about the excluded folder.

The #1 and #2 make it hard for the app to decide when to apply the "workaround", and #3 undermines the workaround per se.

That said, I'm not entirely dismissing the idea, I keep looking around for more ideas.

 

[Morac]

I think the implementation Apple added in Sonoma is just broken. For example I use the AppCleaner app and it can’t delete anything in the ~/Library/Containers folder even though I gave it permission (which doesn’t show up anywhere from what I can tell), it gets an access denied error.

 

[Apple_Robert]

I think the implementation Apple added in Sonoma is just broken. For example I use the AppCleaner app and it can’t delete anything in the ~/Library/Containers folder even though I gave it permission (which doesn’t show up anywhere from what I can tell), it gets an acceded denied error.

I was getting the same error with AppCleaner, even though it had permission and was deleting files although a popup window said it failed. I switched to TrashMe app and haven't had a problem.

 

[TheTerminalMan]

Ok, I've gone down the rabbit-hole on this one. While cleaning up my test script to send to @Oleg K. per his request, I found that not all of the app trees in ~/Library/Containers exhibit the performance issue. I ended up rewriting the script to test every app tree (where possible, as some don't have a Data subdirectory and/or otherwise prevent me from testing).

What I've found using my new test script:

• On any given system, some app trees will show the problem... and some won't.
• Rebooting helps a little, because some app trees no longer show the problem.
  BUT:
• Once an app runs the performance problem with its Container Data tree returns.
• Until/unless Apple provides a fix, no system will ever be completely free of the problem because there's a whole host of Apple services that get started on system boot; once they run their Container Data tree performance goes way down again.
• Interestingly enough, the scripting I have appears to run and is able to walk through several apps' Container data without getting the "Terminal.app would like to access data from other apps" popup. It won't prompt for approval until the script encountes the first "slow" Container tree. It's almost like the protections against accessing another app's Container data don't exist until after the Containerized app has run since the last reboot. Weird.
• And yes, confirming what OP @Morac said: Activity Monitor shows three OS processes (syspolicyd, tccd, and trustd) at the top of the CPU chart when the problem is happening. I'm guessing there's a hook somewhere that sees accesses to the Container Data directories and asks those processes to go think about it for a while.

I'm not sure what DaisyDisk can do to work around the issue, other than provide an option to skip the Container Data directories and/or provide a notice that scan times will be extended because of this. I'm hoping that Apple has a fix for this and it's not a permanent "feature".

I'll restate the concern I alluded to in my initial post: Are the apps whose data is kept in their Library/Containers tree suffering from performance problems caused by this? Or are they somehow exempt because it's their own data and the OS is fast-tracking their access?

@Oleg K., I'll send you a Github link to my scripting so you can play with it some. If there's sufficient interest I'll post the link here unless forum rules prohibit that. (I'm new here and may or may not have actually read them... sorry!).

 

[Czo]

I thought about it, but didn't appeal to me as a good solution, for the following reasons:

1. Not all Sonoma users experience this problem, it only becomes visible perhaps when there are lots of subfolders in ~/Library/Containers. It's impossible to tell in advance whether the scan will stall or not.
2. Even those who experience the problem report that rebooting the Mac temporarily fixes it. (BTW, have you tried this?). Therefore the criterion for applying the workaround becomes more blurry.
3. The ~/Library/Containers includes app data, which is often the source of disk space consumption. Excluding it from the report will potentially add a big chunk of "hidden space", and make it unusable in another way. It will be hard to explain it to the user why he sees the hidden space, and it's easy to forget about the excluded folder.

The #1 and #2 make it hard for the app to decide when to apply the "workaround", and #3 undermines the workaround per se.

That said, I'm not entirely dismissing the idea, I keep looking around for more ideas.

No, not really. With almost 6 days uptime, scanning the whole startup disk takes 19 minutes, and after a reboot this takes exactly the same time too. Meantime syspolicyd, tccd and trustd running a race to win the most cpu time. Under Ventura, this had finished under 2-3 minutes.

So, the ability to manually add thoose folders to an ignore list (in this case it's under two gigabytes), really helps in my situation.

 

[Oleg K.]

Here's the developer of DaisyDisk. I'd like to update you guys. While I'm still waiting for Apple to release a fix for macOS Sonoma, to address the problem of extremely slow access to certain folders, I have just released an update of DaisyDisk (4.26.1) to work around this issue.

This new version can exclude the ~/Library/Containers folder from scanning entirely. This will allow DaisyDisk to complete the scan quickly, but may somewhat increase the "hidden space".

You can download the new verson frem here:
https://daisydiskapp.com/download/DaisyDisk.zip

(or use Check for Updates menu command, or Update from the App Store - depending on where you got the app from)

Here is how to apply the workaround:

Before you click the Scan button or select the Scan as Administrator menu command press and hold the Option key on the keyboard (you can release the key after clicking the Scan button or the menu command).

I hope that helps, let me know if you have questions, and thanks for your feedbacks!

 

[Czo]

Here's the developer of DaisyDisk. I'd like to update you guys. While I'm still waiting for Apple to release a fix for macOS Sonoma, to address the problem of extremely slow access to certain folders, I have just released an update of DaisyDisk (4.26.1) to work around this issue.

This new version can exclude the ~/Library/Containers folder from scanning entirely. This will allow DaisyDisk to complete the scan quickly, but may somewhat increase the "hidden space".

You can download the new verson frem here:
https://daisydiskapp.com/download/DaisyDisk.zip

(or use Check for Updates menu command, or Update from the App Store - depending on where you got the app from)

Here is how to apply the workaround:

Before you click the Scan button or select the Scan as Administrator menu command press and hold the Option key on the keyboard (you can release the key after clicking the Scan button or the menu command).

I hope that helps, let me know if you have questions, and thanks for your feedbacks!

Hi Oleg,

Thanks! That is what i have been waiting for! Now. with this workaround, Daisy Disk can finish under a minute on my 14" MBP.