Help! Hidden Space rapidly taking over on macOS 10.15.1

[MadXD]

Two days ago, my MacBook Pro (15-inch, 2017; macOS 10.15.1 Catalina) told me it was out of storage space. With a 500GB SSD, free space had dropped to 90 MB without me adding any files of significant size.

First step: Purge
I proceeded to scan through and remove unnecessary files, clearing about 50 GB. Within the next hour, the MacBook's free space continued to drop from 50GB to 1GB—without any interaction by me.

Second Step: Find the large files
Next, I searched through my SSD to see where the space was being used. The first oddity is that my Users folder shows a size of 475GB. Inside, I found only 217 GB of files [home folder @ 217 GB + Shared folder @ 8MB]. Note that my hidden files are shown and no hidden files are in my Users folder.

Third Step: Self-Diagnosis

1. Disk Utility First Aid [in recovery mode] - Data and System partitions OK
2. Daisy Disk [in administrator mode with full disk access] - Found 221GB of "hidden space"

Within this "Hidden Space" lies my problem. Neither Finder nor Daisy Disk can reveal the contents of this hidden space.
3) Malwarebytes [with full disk access] - Zero issues or vulnerabilities

Fourth Step: Find Help

1. AppleCare Support - I explained and demonstrated the issue via screen sharing. Case was escalated to a senior advisor who has filed it with engineering. I provided Apple-collected diagnostic files and screenshots of system status and file sizes. Now I have to wait. I should hear back Monday about the response from macOS engineering.
2. Reddit - No responses
3. MacRumors Forums - Is anyone else experiencing something like this? I haven't been able to find a similar issue online.

It has nearly halted the ability to use my MacBook Pro. The lack of free disk space chokes applications that need it. My available disk space is currently fluctuating up and down, though not above 2GB (I deleted the 6.5GB Xcode app 30 minutes ago). When free space drops too low many apps and services won't run, such as: Setapp (or apps installed under it), Mail, iCloud documents, and macOS continually asks to close applications due to low memory.

Update: Four hours later, I woke up my "sleeping" MacBook Pro to find free space dropped to less than 20MB. The MacBook is warm and the fans are running.

Hidden Space is actively and aggressively taking over my Mac!

NOTE: I was on the Catalina Public Beta but have been off since the public release. The issue began 4-5 days after I updated to macOS 10.15.1.

 

[StellarVixen]

Can you try OmniDiskSweeper and report back?


It's the free little utility that helped me track down the culprits. No, when I say free, I mean free.

 

[MadXD]

Can you try OmniDiskSweeper and report back?


It's the free little utility that helped me track down the culprits. No, when I say free, I mean free.

Trying now!
[automerge]1573381709[/automerge]
For the record, I do not use TimeMachine and the following tip to remove snapshots did not help—I don't have any snapshots.

Disk full: Re-claim disk free space after Catalina install

After installing Catalina and you go to About This Mac-Storage, your disk might be nearly full even though you actually don't have that many files to fill up your disk. Claim your free space back by doing the following: 1) Open the Terminal app 2) Type: tmutil listlocalsnapshots...

forums.macrumors.com

 

[MadXD]

Can you try OmniDiskSweeper and report back?


It's the free little utility that helped me track down the culprits. No, when I say free, I mean free.

Unfortunately OmniDisk Sweeper doesn't even see the hidden space. It only reports 263.8 GB of used space.

Meanwhile, both macOS System Info and Daisy Disk report 498 GB of used space.
System Info reports 292 GB of "Other" space.

Daisy Disk reports 235 GB of "Hidden Space".

Deleting any file of 'x' size causes reported "Other" and "Hidden Space" above to increase by 'x' size.

 

[StellarVixen]

Well, continue clicking through the directories, until you find ones taking most storage.


See:

As long as there is arrow beside directory name, you can open it further.

 

[MadXD]

Nothing listed in OmniDiskSweeper is an issue. I already know the size, location and purpose of every folder and file it found.

I need to find what Omni isn't seeing... something hidden and using 234.5 GB of my 500 GB SSD.

 

[StellarVixen]

You haven't told us do you have Time Machine turned on?

 

[MadXD]

For the record, I do not use TimeMachine and the following tip to remove snapshots did not help—I don't have any snapshots.

Disk full: Re-claim disk free space after Catalina install

After installing Catalina and you go to About This Mac-Storage, your disk might be nearly full even though you actually don't have that many files to fill up your disk. Claim your free space back by doing the following: 1) Open the Terminal app 2) Type: tmutil listlocalsnapshots...

forums.macrumors.com

No, I do not use Time Machine

 

[MadXD]

Actually, I posted specifically that I don't use time machine... but the post has this above it:

"This message is awaiting moderator approval, and is invisible to normal visitors."

I have already attempted to locate and remove built-up Time Machine snapshots. Unfortunately none exit, at least not that I could find using tmutil listlocalsnapshots /System/Volumes/Data

 

[dsemf]

Actually, I posted specifically that I don't use time machine... but the post has this above it:

"This message is awaiting moderator approval, and is invisible to normal visitors."

I have already attempted to locate and remove built-up Time Machine snapshots. Unfortunately none exit, at least not that I could find using tmutil listlocalsnapshots /System/Volumes/Data

I prefer to use terminal commands. They are not as user friendly but don't have assumptions built in.

The first one shows the APFS structure of your boot disk.

yyyy:~ xxx$ diskutil apfs list
APFS Container (1 found)
|
+-- Container disk1 3BEAE5D6-797A-4B0B-B86E-0DE7437ADB3B
    ====================================================
    APFS Container Reference:     disk1
    Size (Capacity Ceiling):      500068036608 B (500.1 GB)
    Capacity In Use By Volumes:   189918261248 B (189.9 GB) (38.0% used)
    Capacity Not Allocated:       310149775360 B (310.1 GB) (62.0% free)
    |
    +-< Physical Store disk0s2 58F4147A-1E3B-4808-AB9E-6662BFABC7C9
    |   -----------------------------------------------------------
    |   APFS Physical Store Disk:   disk0s2
    |   Size:                       500068036608 B (500.1 GB)
    |
    +-> Volume disk1s2 7808D72D-9C79-4996-BD9D-0A73876FF5C7
    |   ---------------------------------------------------
    |   APFS Volume Disk (Role):   disk1s2 (Preboot)
    |   Name:                      Preboot (Case-insensitive)
    |   Mount Point:               Not Mounted
    |   Capacity Consumed:         85467136 B (85.5 MB)
    |   FileVault:                 No
    |
    +-> Volume disk1s3 0256B625-8294-4038-8816-546629693DBC
    |   ---------------------------------------------------
    |   APFS Volume Disk (Role):   disk1s3 (Recovery)
    |   Name:                      Recovery (Case-insensitive)
    |   Mount Point:               /Volumes/Recovery
    |   Capacity Consumed:         528543744 B (528.5 MB)
    |   FileVault:                 No
    |
    +-> Volume disk1s4 B45C2266-58E2-403B-8E2D-9CF4280AEE45
    |   ---------------------------------------------------
    |   APFS Volume Disk (Role):   disk1s4 (VM)
    |   Name:                      VM (Case-insensitive)
    |   Mount Point:               /private/var/vm
    |   Capacity Consumed:         2148552704 B (2.1 GB)
    |   FileVault:                 No
    |
    +-> Volume disk1s5 65B255B6-EB33-4BCE-A6C8-B62A5BE88BF6
    |   ---------------------------------------------------
    |   APFS Volume Disk (Role):   disk1s5 (Data)
    |   Name:                      Catalina HD - Data (Case-insensitive)
    |   Mount Point:               /System/Volumes/Data
    |   Capacity Consumed:         176120127488 B (176.1 GB)
    |   FileVault:                 Yes (Unlocked)
    |
    +-> Volume disk1s6 A36114C6-61B1-4C82-B113-C051EB3B5109
        ---------------------------------------------------
        APFS Volume Disk (Role):   disk1s6 (System)
        Name:                      Catalina HD (Case-insensitive)
        Mount Point:               /
        Capacity Consumed:         10873937920 B (10.9 GB)
        FileVault:                 Yes (Unlocked)

This shows that I have one 500GB Container with 190GB used and 310GB free. The container contains 5 Volumes. The System volume is 11GB and the Data volume is 176GB.

If you have granted Full Disk Access to Terminal using System Preferences >> Security and Privacy, Privacy tab, you can use the following command to get a summary of the disk usage by location. Since the majority of the data usage wll be in the Data volume, make note of the Mount Point. It will normally be /System/Volumes/Data.

% sudo du -d 1 -x -c -g /System/Volumes/Data
Password:
0    /System/Volumes/Data/sw
1    /System/Volumes/Data/home
1    /System/Volumes/Data/usr
2    /System/Volumes/Data/.Spotlight-V100
2    /System/Volumes/Data/Library
0    /System/Volumes/Data/.Trashes
3    /System/Volumes/Data/System
0    /System/Volumes/Data/mnt
1    /System/Volumes/Data/.fseventsd
du: /System/Volumes/Data/private/var/db/fpsd/dvp: Operation not permitted
...
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/0/com.apple.LaunchServices.dv: Operation not permitted
4    /System/Volumes/Data/private
1    /System/Volumes/Data/.DocumentRevisions-V100
124    /System/Volumes/Data/Users
7    /System/Volumes/Data/Applications
0    /System/Volumes/Data/opt
0    /System/Volumes/Data/Volumes
0    /System/Volumes/Data/.TemporaryItems
0    /System/Volumes/Data/cores
140    /System/Volumes/Data
140    total

There will be a number of errors for /private/var/ locations. Ignore these.

To drill down, extend the path at the end of the du command

Since Data actually has 176GB, the other 36GB is most likely in the /private/var/ area or is reserved by the operating system for pending usage.

Doing the above may do nothing more than confirm what the other tools have identified but it does provide a different approach.

DS

 

[mpainesyd]

There is a similar discussion about High Sierra here:

High Sierra System Files eating up 400gb of space ... what gives??

I don't use Time Machine. Users and Photos are about right.

forums.macrumors.com

Several people found backup apps like Time Machine and Carbon Copy Cloner were the culprits, particularly if there were corrupted files. Same for Spotlight.
Also a useful discussion about AFPS using purgeable space here

How to free up disk space in macOS High Sierra

DaisyDisk now lets you delete Purgeable Space on your disk, the 'dark matter' storage that can fill up to 80% of your Mac's storage without you even knowing.

www.cultofmac.com

 

[mellie13]

I have the same issue! I thought it was a perl script I was working on but that doesn't seem to be the case. This is definitely temporary storage as I also see it as categorized under other and when I reboot, it magically reclaims the space. I am running teamViewer and vmWare, and I am wondering if they are the culprits, are you running either of there tools? I have now resorted to running du hourly and diffing the files. The changes are too many so I am narrowing down the folders that change the most but that is a very difficult approach

 

[Rockabilico]

Same issue here!! Was there any response from the Apple?
Did anyone found a solution?

 

[Will Martins]

Hi @madizin Have you received any return from apple? I'm at a very similar situation. To me appears that there are some problem with apfs and the new mount points. I notices that the storage as hidden, is the same siza of /Users , and also noticed that inside /System/Volumes/Data has a /Users folders also. When I check with "du" the / mount point , it's give me almost the size of the disk. In my case, the half of the disk is full with hidden files. Tried everythig. The last option is resinstall everything cleaning all disk ( that I would like to aviod ) , or simply reinstall macos ( impossible bc have just 4B free ) Tell us how did you solved please! Cheers

 

[bousozoku]

Have you tried OnyX to help you clean up what might otherwise be inaccessible?

I've always found it useful, especially in the early days of Mac OS X, when it didn't always clean up after itself.

You should do some research to make sure you're okay with the outcome, but it was often able to quickly fix some problems that would have taken a lot of manual work.

 

[Will Martins]

Have you tried OnyX to help you clean up what might otherwise be inaccessible?

I've always found it useful, especially in the early days of Mac OS X, when it didn't always clean up after itself.

You should do some research to make sure you're okay with the outcome, but it was often able to quickly fix some problems that would have taken a lot of manual work.

I've tried onyx already, daisy, diskinventory, cleanmac, EVERYTHING. There is no snapshots of tm . Everytime I delete something, the "others" space ( hidden space ) increases. I did a lot of research . Right now, I'm doing 2 things: backingup important files manually, and trying to make a timecapsule remote backup . Than I will try to repair the disk1 in safe mode, and if not work, reformat everything. Hope that timecapsule works to recorver from the safe mode after full erease.

 

[bousozoku]

I've tried onyx already, daisy, diskinventory, cleanmac, EVERYTHING. There is no snapshots of tm . Everytime I delete something, the "others" space ( hidden space ) increases. I did a lot of research . Right now, I'm doing 2 things: backingup important files manually, and trying to make a timecapsule remote backup . Than I will try to repair the disk1 in safe mode, and if not work, reformat everything. Hope that timecapsule works to recorver from the safe mode after full erease.

It's sad to be at that extreme. I've not been there since the early days of Mac OS X, when the installer needed to be run 2-3 times to install everything. On the other hand, Windows 10 regularly does ugly things when Microsoft updates it. I haven't had full use of that machine since December, maybe.

 

[dsemf]

Hi @madizin Have you received any return from apple? I'm at a very similar situation. To me appears that there are some problem with apfs and the new mount points. I notices that the storage as hidden, is the same siza of /Users , and also noticed that inside /System/Volumes/Data has a /Users folders also. When I check with "du" the / mount point , it's give me almost the size of the disk. In my case, the half of the disk is full with hidden files. Tried everythig. The last option is resinstall everything cleaning all disk ( that I would like to aviod ) , or simply reinstall macos ( impossible bc have just 4B free ) Tell us how did you solved please! Cheers

/Users and /System/Volumes/Data/Users are the same location. /Users is a special link to the real location.

When I have questions about space usage, I use terminal. The output has been edited to reduce the volume.

cd /System/Volumes/Data
sudo du -d 1 -x -c -g .
1    ./.DocumentRevisions-V100
178    ./Users
18    ./Applications
0    ./opt
0    ./Volumes
0    ./.TemporaryItems
0    ./cores
209    .
209    total

df -h
Filesystem      Size   Used  Avail Capacity   iused      ifree %iused  Mounted on
/dev/disk1s6    466Gi   10Gi  220Gi     5%    487665 4882989255    0%   /
devfs           190Ki  190Ki    0Bi   100%       656          0  100%   /dev
/dev/disk1s5    466Gi  230Gi  220Gi    52%   1757443 4881719477    0%   /System/Volumes/Data
/dev/disk1s4    466Gi  5.0Gi  220Gi     3%         6 4883476914    0%   /private/var/vm

I can see that I have 10GB of system data (the OS) and 209GB of data which uses 230GB on the data volume. The du command does not authorization to look at everything in /private/var so that might be part of the missing data.

If I need more detail, I can extent the . to ./Users, etc.

DS

 

[bousozoku]

Those of you having a problem, was your drive converted to APFS automatically, or was it formatted as APFS and your data restored manually?

I noticed problems with the conversion process on my hard drive, initially, but I formatted an SSD as APFS and had zero drive-related problems after that.

 

[rmfrank]

It's been a while since the last activity, but I've just faced the same problem. Using the terminal and the sudo du -d 1 -x -c -g /System/Volumes/Data/... command (drilling down the path with the largest size), I ended up in my library (of my account) and in the mail configuration. There is a log directory there, which was filling up.

So, in my case, the culprit is Apple Mail - but I don't yet know why that is filling the log file like mad.

The path, in my case is: /System/Volumes/Data/Users/<user>/Library/Containers/com.apple.mail/Data/Library/Logs/Mail

Within that directory, it was a log file (.txt) from the mail connected to a Microsoft exchange account ... (214Gb)
But this was not the only big one, all imap accounts are also producing large log files.

 

[JackieTreehorn]

Let me just add to this, but I have the same problems on three different machines (2019 MBA, 2015 iMac, and 2019 16" MBP).

It's horrible, hidden space take up at least 70% of the SSD storage on all three machines without any clue on what to do to get rid off it. Apart from maybe a clean install.

 

[Heath Close]

I'm in the same boat... will comb through this again because I have suspected in the past that the time machine might have something to do with it.

I use grand perspective to find large files and I literally just cleaned off like 50 gigs 2 days ago. They are back.

I'll go back up and try some of the solutions for the time machine fix first.

 

[Heath Close]

as you can see... 50 gb in "misc"

 

[Heath Close]

I thought I had a virus. I even tried to reinstall but it kept failing. I tried changing virus programs. Today, I cleaned it up in 2 steps thanks to DaisyDisk. Thank you to those who pointed me to it!

Is there a way to shut snapshots off? That was the culprit.

 

[MadXD]

@Will Martins @mellie13 @Rockabilico @dsemf @Heath Close

I apologize to anyone who started helping me; or anyone looking for more insight for similar issues.

After starting this thread, MacRumors blocked my account because I had created two (an unintentional side-effect of wanting a new username). It more than a week to have my account re-activated. In the meantime, I couldn't respond or update with the solution.

Due to overly-protective and difficult-to-contact MacRumors admins, it was other MacRumors users and site guests who suffered. I ended up moving the conversation to Reddit, where I received zero responses, input or signs that my experience was able to help anyone else.

Conclusion below: