360 GB of Ghost Data/Unrecovered Space?

[zedsdeadbaby]

Hey everyone... I'm seeing some weird storage issues on a new MacBook Pro I bought in September.

I deleted 100+ GBs of data last week. Checked space the next day and noticed the numbers didn't seem right... So, I decided to before/after screenshot what Disk Utility was reporting as free space to see if I miscalculated... I moved 16 GB, deleted the folder then emptied the trash. Trash said it deleted the files but Disk Utility showed only a 75 MB difference.

Since then I've been checking Disk Utility and finder while deleting files and same story... Space is either not being recovered or the file index isn't updating...

Anyway long story short, I decided to run a tree mapping apps today and something weird is going on for sure... Both Grand Perspective and Disk Inventory X show my volume as 3.46 TB instead of 4 TB. Even weirder is that Grand Perspective shows a huge block of grey in between the empty space if I lay all of the tree maps next to one another in photoshop. Basically it looks like there's 360 GB block of unrecovered disk space causing the Volume to be calculated incorrectly.

Been going back and forth with support but thought I'd see if anyone else has seen this, and make sure the tree map displays what I think it does... (Plus the Volume Size... just plain weird...)

My system Volume.

Project files

Samples and Assets volume:

All volumes laid next to one another...

 

[Thysanoptera]

3.64 TB is the usable space of a 4 TB drive unfortunately. Software calculates the space using binary system, and marketing people the decimal one, the difference is between 4*2^40 and 4*10^12. Technically the usable space is measured in TiB, so 4TB~3.64TiB

The grey space has something to do with local timemachine snapshots, the OS takes care of it itself, but you can turn it off. I had something similar, couple hundred GB as 'system', after I turned off local snapshots it went down to 20.

 

[Fishrrman]

OP:

Graphical "displays of data" (like you get from Grand Perspective) don't help much, IMO.

For a better indication of what's actually "eating up that space", try DiskWave, free from here:
https://diskwave.barthe.ph

Once you have it, open it.
Go to preferences and choose to make normally-invisible files visible.

Then, just look around.
You should see what to do next...

 

[iMacDragon]

What format is the drive, apfs or hfs. Apfs certainly seems to keep a lot of fully hidden stuff in snapshots for time machine which only get freed up when necessary it seems.

 

[zedsdeadbaby]

3.64 TB is the usable space of a 4 TB drive unfortunately. Software calculates the space using binary system, and marketing people the decimal one, the difference is between 4*2^40 and 4*10^12. Technically the usable space is measured in TiB, so 4TB~3.64TiB

The grey space has something to do with local timemachine snapshots, the OS takes care of it itself, but you can turn it off. I had something similar, couple hundred GB as 'system', after I turned off local snapshots it went down to 20.

Thanks for explaining. Very Insightful, kind of a bummer, but insightful... I now see why G.P. lets you choose binary vs Logical.

In terms of TM I don't use it, I've always used CCC instead since it's bootable. I have run TM three times over the last few weeks for Apple diagnostics but only on the system volume. Can't imagine a few local snapshots would take up that much space...

The other thing is that it still doesn't explain why Disk Utility isn't showing recovered space despite deleting many gigs at a time... I've safe booted, cleared caches, all the usual stuff. D.U. doesn't show space being recovered.

OP:

Graphical "displays of data" (like you get from Grand Perspective) don't help much, IMO.

For a better indication of what's actually "eating up that space", try DiskWave, free from here:
https://diskwave.barthe.ph

Once you have it, open it.
Go to preferences and choose to make normally-invisible files visible.

Then, just look around.
You should see what to do next...

Thanks for the tip. Will download in a sec and see what shows up there.

What format is the drive, apfs or hfs. Apfs certainly seems to keep a lot of fully hidden stuff in snapshots for time machine which only get freed up when necessary it seems.

The drive is APFS. No HFS partitions, just two APFS containers for the project files and assets. Again though, not normally using TM, just CCC to two external disks. A scrap SSD for a bootable system clone and a platter drive for the other two volumes.

 

[zedsdeadbaby]

Just wanted to update this... I got the numbers in Finder and GrandPerspetive to match within a few megs.

As Thysanoptera pointed out it was a matter of changing binary to decimal. After changing to decimal GP, Finder, and Disk Utility all show the same amount of free space, folder sizes match, and all three apps (other than Disk Utility) reinforce one another's numbers...

What I've found is that the system volume updates in tree view apps to reflect the deleted space, Disk Utility however only reports fractions of space are being recovered when emptying the trash, and volume sizes are way off...

During several screen recordings I caught Disk Utility recovered only 3 GB when deleting 40 GB. Another showed about 2 GB recovered when deleting 27 GB.

Reboots did not change the size recovered and volume sizes reported in D.U. vs GP (and Disk Wave) are hundreds of gigs apart and correlate the size of ghost space shown in GP... In total 580+ GB of orphaned data.

Not sure if it's a T2/BridgeOS issue or APFS issue.. Either way, I've seen ghost data before but nothing remotely close to this...

 

[Thysanoptera]

CCC also makes local snapshots, you can disable this function and delete snapshots from within CCC GUI. And sometimes the reported size is wrong, so just delete all you can see.

 

[zedsdeadbaby]

CCC also makes local snapshots, you can disable this function and delete snapshots from within CCC GUI. And sometimes the reported size is wrong, so just delete all you can see.

Checked, CCC snapshots were of course enabled. After reviewing some Bombich links this also explains why I was only seeing partial space recover when emptying the trash... Thanks again.

Curious about one other thing... Any idea if these Disk Utility errors are related?

warning: apfs_num_other_fsobjects (64) is not valid (66)
...
Checking the snapshot metadata tree.
warning: object (oid 0x21adab64): Unable to mark physical extent range (0x21adab64 + 111) allocated for space verification
warning: object (oid 0x21adabd3): Unable to mark physical extent range (0x21adabd3 + 10) allocated for space verification
warning: object (oid 0x21adabdd): Unable to mark physical extent range (0x21adabdd + 110) allocated for space verification
warning: object (oid 0x21adac4b): Unable to mark physical extent range (0x21adac4b + 4096) allocated for space verification

 

[Thysanoptera]

No idea, Apple doesn't share too much information about AFPS, but that does look like some mess left over by CCC created snapshots. Or maybe the old timemachine ones. AFPS supports snapshots so I guess they both use native files system snapshots instead of creating something in own format and some stuff could have been mixed up when you switched from TM to CCC. Maybe you can try booting in recovery mode and run diskutil there. Also check with tmutil if there is anything left from timemachine.

 

[zedsdeadbaby]

Just an update.. Deleting snapshots recovered the space, 1.2 TB free now.
The Disk Utility warnings are still there after deleting them unfortunately, no answer yet from engineering about them...