Storage in /System/Volumes/Data doesn't add up ("Other" taking a lot)

[astroboylrx]

I'm confused by the storage stats. I have a macOS Big Sur under APFS (~736 Gi with 578 Gi used) and also a BootCamp Windows (~189 Gi with ~143 Gi used) alongside. "System Information" says I only have ~156G left (= 143Gi) left in macOS, which is consistent with the command df -h as shown below. However, the content inside /System/Volumes/Data doesn't add up to 578 Gi. Using the command du -chd 1 only gives merely ~398 Gi (180 Gi off) -- see the code block below.

The result described above is without mounting the BootCamp partition. If I mount on the BootCamp partition, there is just 143 Gi more data under /System/Volumes/Data/Volumes, which makes /System/Volumes/Data 541 Gi in total, closer to 578 Gi but still 37 Gi off.

What's happening here? Where does the 37 Gi come from? In those "Operation not permitted" files?
Why does macOS take into account the storage used in BootCamp? If not, where is the 178 Gi off from?
I've also tried GrandPerspective, which told me there is 182 Gi miscellaneous used space, which seems to be consistent with the 180 Gi off...but where are those files that occupy space?

Thanks in advance.

➜  /System/Volumes/Data $ df -h
Filesystem       Size   Used  Avail Capacity iused      ifree %iused  Mounted on
/dev/disk1s5s1  736Gi   14Gi  143Gi     9%  567557 7714276163    0%   /
devfs           199Ki  199Ki    0Bi   100%     690          0  100%   /dev
/dev/disk1s4    736Gi   20Ki  143Gi     1%       2 7714843718    0%   /System/Volumes/VM
/dev/disk1s2    736Gi  315Mi  143Gi     1%    1179 7714842541    0%   /System/Volumes/Preboot
/dev/disk1s6    736Gi  2.1Mi  143Gi     1%      16 7714843704    0%   /System/Volumes/Update
/dev/disk1s1    736Gi  578Gi  143Gi    81% 5908574 7708935146    0%   /System/Volumes/Data
map auto_home     0Bi    0Bi    0Bi   100%       0          0  100%   /System/Volumes/Data/home

➜  /System/Volumes/Data $ sudo du -chd 1 | sort -h
Password:
du: ./Library/Caches/com.apple.aned: Operation not permitted
du: ./private/var/networkd/db: Operation not permitted
du: ./private/var/db/appinstalld: Operation not permitted
du: ./private/var/db/fpsd/dvp: Operation not permitted
du: ./private/var/db/installcoordinationd: Operation not permitted
du: ./private/var/db/oah: Operation not permitted
du: ./private/var/db/lockdown: Operation not permitted
du: ./private/var/db/DifferentialPrivacy: Operation not permitted
du: ./private/var/db/MobileIdentityService: Operation not permitted
du: ./private/var/db/searchparty: Operation not permitted
du: ./private/var/db/CoreDuet/Knowledge: Operation not permitted
du: ./private/var/db/ConfigurationProfiles/Store: Operation not permitted
du: ./private/var/folders/9j/k6l8ydqx1m9_wbl3ygzxxsd40000gl/0/com.apple.LaunchServices.dv: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/0/SafariFamily: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/0/com.apple.ScreenTimeAgent/Store: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/0/com.apple.lockoutagent: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/0/com.apple.progressd/ClassKit: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/0/com.apple.Safari/SafariFamily: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/0/com.apple.LaunchServices.dv: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/0/dmd: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/0/com.apple.nsurlsessiond: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/0/com.apple.SharedWebCredentials: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/0/com.apple.routined: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/C/com.apple.WebKit.WebContent.Sandbox: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/C/com.apple.QuickLook.thumbnailcache: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/C/com.apple.WebKit.Networking.Sandbox: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/C/com.apple.quicklook.ThumbnailsAgent/com.apple.QuickLook.thumbnailcache: Operation not permitted
du: ./private/var/folders/91/2d4txqms4dq7gqm8sdd1ts7w0000gq/0/com.apple.Safari/SafariFamily: Operation not permitted
du: ./private/var/folders/91/2d4txqms4dq7gqm8sdd1ts7w0000gq/0/com.apple.LaunchServices.dv: Operation not permitted
du: ./private/var/folders/91/2d4txqms4dq7gqm8sdd1ts7w0000gq/0/com.apple.nsurlsessiond: Operation not permitted
du: ./private/var/folders/91/2d4txqms4dq7gqm8sdd1ts7w0000gq/0/com.apple.routined: Operation not permitted
du: ./private/var/folders/91/2d4txqms4dq7gqm8sdd1ts7w0000gq/C/com.apple.WebKit.WebContent.Sandbox: Operation not permitted
du: ./private/var/folders/91/2d4txqms4dq7gqm8sdd1ts7w0000gq/C/com.apple.WebKit.Networking.Sandbox: Operation not permitted
du: ./private/var/folders/zz/zyxvpxvq6csfxvn_n00000sm00006d/0: Operation not permitted
du: ./private/var/folders/zz/zyxvpxvq6csfxvn_n00000sm00006d/C: Operation not permitted
du: ./private/var/folders/zz/zyxvpxvq6csfxvn_n00000y800007k/0/com.apple.nsurlsessiond: Operation not permitted
du: ./private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.nsurlsessiond: Operation not permitted
du: ./private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.routined: Operation not permitted
du: ./private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/0/com.apple.LaunchServices.dv: Operation not permitted
du: ./private/var/folders/xm/56ppty1555l3d4sjdb6z5wzr0000gp/0/com.apple.LaunchServices.dv: Operation not permitted
du: ./private/var/protected/trustd/private: Operation not permitted
du: ./Users/my_user/Library/VoiceTrigger/SAT: Operation not permitted
du: ./Users/my_user/Library/Caches/VoiceTrigger/SATUpdate: Operation not permitted
  0B    ./.PKInstallSandboxManager
  0B    ./.PKInstallSandboxManager-SystemSoftware
  0B    ./.TemporaryItems
  0B    ./.Trashes
  0B    ./.vol
  0B    ./Volumes
  0B    ./cores
  0B    ./mnt
  0B    ./sw
1.0K    ./home
5.3M    ./.PreviousSystemInformation
6.0M    ./usr
155M    ./.Spotlight-V100
243M    ./.DocumentRevisions-V100
258M    ./boot
403M    ./.fseventsd
7.2G    ./System
8.7G    ./private
13G    ./opt
33G    ./Library
81G    ./Applications
255G    ./Users
398G    .
398G    total

------ After mounting BootCamp -----

➜  /System/Volumes/Data $ df -h
Filesystem       Size   Used  Avail Capacity iused      ifree %iused  Mounted on
/dev/disk1s5s1  736Gi   14Gi  143Gi     9%  567557 7714276163    0%   /
devfs           199Ki  199Ki    0Bi   100%     690          0  100%   /dev
/dev/disk1s4    736Gi   20Ki  143Gi     1%       2 7714843718    0%   /System/Volumes/VM
/dev/disk1s2    736Gi  315Mi  143Gi     1%    1179 7714842541    0%   /System/Volumes/Preboot
/dev/disk1s6    736Gi  2.1Mi  143Gi     1%      16 7714843704    0%   /System/Volumes/Update
/dev/disk1s1    736Gi  578Gi  143Gi    81% 5910893 7708932827    0%   /System/Volumes/Data
map auto_home     0Bi    0Bi    0Bi   100%       0          0  100%   /System/Volumes/Data/home
/dev/disk0s3    186Gi  143Gi   43Gi    77%  303109   45677563    1%   /Volumes/BOOTCAMP

➜  /System/Volumes/Data sudo du -chd 1 | sort -h
Password:
du: ./Library/Caches/com.apple.aned: Operation not permitted
du: ./private/var/networkd/db: Operation not permitted
du: ./private/var/db/appinstalld: Operation not permitted
du: ./private/var/db/fpsd/dvp: Operation not permitted
du: ./private/var/db/installcoordinationd: Operation not permitted
du: ./private/var/db/oah: Operation not permitted
du: ./private/var/db/lockdown: Operation not permitted
du: ./private/var/db/DifferentialPrivacy: Operation not permitted
du: ./private/var/db/MobileIdentityService: Operation not permitted
du: ./private/var/db/searchparty: Operation not permitted
du: ./private/var/db/CoreDuet/Knowledge: Operation not permitted
du: ./private/var/db/ConfigurationProfiles/Store: Operation not permitted
du: ./private/var/folders/9j/k6l8ydqx1m9_wbl3ygzxxsd40000gl/0/com.apple.LaunchServices.dv: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/0/SafariFamily: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/0/com.apple.ScreenTimeAgent/Store: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/0/com.apple.lockoutagent: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/0/com.apple.progressd/ClassKit: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/0/com.apple.Safari/SafariFamily: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/0/com.apple.LaunchServices.dv: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/0/dmd: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/0/com.apple.nsurlsessiond: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/0/com.apple.SharedWebCredentials: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/0/com.apple.routined: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/C/com.apple.WebKit.WebContent.Sandbox: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/C/com.apple.QuickLook.thumbnailcache: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/C/com.apple.WebKit.Networking.Sandbox: Operation not permitted
du: ./private/var/folders/8n/91kyfm9d5vb2s6ft_l4wrt480000gn/C/com.apple.quicklook.ThumbnailsAgent/com.apple.QuickLook.thumbnailcache: Operation not permitted
du: ./private/var/folders/91/2d4txqms4dq7gqm8sdd1ts7w0000gq/0/com.apple.Safari/SafariFamily: Operation not permitted
du: ./private/var/folders/91/2d4txqms4dq7gqm8sdd1ts7w0000gq/0/com.apple.LaunchServices.dv: Operation not permitted
du: ./private/var/folders/91/2d4txqms4dq7gqm8sdd1ts7w0000gq/0/com.apple.nsurlsessiond: Operation not permitted
du: ./private/var/folders/91/2d4txqms4dq7gqm8sdd1ts7w0000gq/0/com.apple.routined: Operation not permitted
du: ./private/var/folders/91/2d4txqms4dq7gqm8sdd1ts7w0000gq/C/com.apple.WebKit.WebContent.Sandbox: Operation not permitted
du: ./private/var/folders/91/2d4txqms4dq7gqm8sdd1ts7w0000gq/C/com.apple.WebKit.Networking.Sandbox: Operation not permitted
du: ./private/var/folders/zz/zyxvpxvq6csfxvn_n00000sm00006d/0: Operation not permitted
du: ./private/var/folders/zz/zyxvpxvq6csfxvn_n00000sm00006d/C: Operation not permitted
du: ./private/var/folders/zz/zyxvpxvq6csfxvn_n00000y800007k/0/com.apple.nsurlsessiond: Operation not permitted
du: ./private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.nsurlsessiond: Operation not permitted
du: ./private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.routined: Operation not permitted
du: ./private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/0/com.apple.LaunchServices.dv: Operation not permitted
du: ./private/var/folders/xm/56ppty1555l3d4sjdb6z5wzr0000gp/0/com.apple.LaunchServices.dv: Operation not permitted
du: ./private/var/protected/trustd/private: Operation not permitted
du: ./Users/my_user/Library/VoiceTrigger/SAT: Operation not permitted
du: ./Users/my_user/Library/Caches/VoiceTrigger/SATUpdate: Operation not permitted
  0B    ./.PKInstallSandboxManager
  0B    ./.PKInstallSandboxManager-SystemSoftware
  0B    ./.TemporaryItems
  0B    ./.Trashes
  0B    ./.vol
  0B    ./cores
  0B    ./mnt
  0B    ./sw
1.0K    ./home
5.3M    ./.PreviousSystemInformation
6.0M    ./usr
243M    ./.DocumentRevisions-V100
258M    ./boot
403M    ./.fseventsd
1.8G    ./.Spotlight-V100
7.2G    ./System
8.7G    ./private
13G    ./opt
33G    ./Library
81G    ./Applications
142G    ./Volumes
255G    ./Users
541G    .
541G    total

 

[gilby101]

➜ /System/Volumes/Data $ df -h
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/disk1s5s1 736Gi 14Gi 143Gi 9% 567557 7714276163 0% /
devfs 199Ki 199Ki 0Bi 100% 690 0 100% /dev
/dev/disk1s4 736Gi 20Ki 143Gi 1% 2 7714843718 0% /System/Volumes/VM
/dev/disk1s2 736Gi 315Mi 143Gi 1% 1179 7714842541 0% /System/Volumes/Preboot
/dev/disk1s6 736Gi 2.1Mi 143Gi 1% 16 7714843704 0% /System/Volumes/Update
/dev/disk1s1 736Gi 578Gi 143Gi 81% 5910893 7708932827 0% /System/Volumes/Data
map auto_home 0Bi 0Bi 0Bi 100% 0 0 100% /System/Volumes/Data/home
/dev/disk0s3 186Gi 143Gi 43Gi 77% 303109 45677563 1% /Volumes/BOOTCAMP

Firstly, diskutil list is more usual because macOS mostly counts GB. Nevertheless, I see you have:

APFS container/partition (736) + Bootcamp partition (186) = 932 Gi. I assume that is correct (1000 GB).

Your APFS container has:
disk1s5 (14) + disk1s1 (578) + available (143) + a few bits = 735 Gi (= 736 within rounding error).

That all adds up correctly. I wouldn't look any further.

/System/Volumes is a mishmash of real devices and pretend devices constructed by macOS. I don't think it is worth pursuing differences between it and df or diskutil.

You might want to explore all of / which will include both disk1s1 and disk1s5 and compare that with 14+578Gi. I find Daisy Disk (scanning as administrator) gives pretty accurate numbers for what is in the macOS system disk, though you have to watch out for the VM volume also appearing /var/db/VM and the uncertainties over purgeable space and TM snapshots.

 

[astroboylrx]

Firstly, diskutil list is more usual because macOS mostly counts GB. Nevertheless, I see you have:

APFS container/partition (736) + Bootcamp partition (186) = 932 Gi. I assume that is correct (1000 GB).

Your APFS container has:
disk1s5 (14) + disk1s1 (578) + available (143) + a few bits = 735 Gi (= 736 within rounding error).

That all adds up correctly. I wouldn't look any further.

/System/Volumes is a mishmash of real devices and pretend devices constructed by macOS. I don't think it is worth pursuing differences between it and df or diskutil.

You might want to explore all of / which will include both disk1s1 and disk1s5 and compare that with 14+578Gi. I find Daisy Disk (scanning as administrator) gives pretty accurate numbers for what is in the macOS system disk, though you have to watch out for the VM volume also appearing /var/db/VM and the uncertainties over purgeable space and TM snapshots.

Thanks a lot for the reply. Yeah, the total number seems right. But there is only 398 Gi data inside /dev/disk1s1 or /System/Volumes/Data when counting all the files. However, df -h says the used is 578 Gi. That's the main discrepancy I want to figure out.

Also, I'm sure that I don't have local TM snapshots...

---
After writing the last sentence, I wanted to make sure I disabled the automatic backup in Time Machine. Yeah, it was disabled. However, I soon found out Time Machine still generates local snapshots no matter how and they even removed the disablelocal verb in the tmutil command!
It is indeed the local snapshots of the Time Machines that occupied over 180 Gi of space. Feel like I wasted a few of my afternoons to chase down this rabbit hole.