Lost files and permissions after out of memory crash

[t2jd1967]

Yesterday, something weird happend on my M2 MBP 14 (16/10/1TB). I was running some CPU intensive multithreaded Python code on two terminal sessions when I experienced some serious slowing down and eventually a popup came on the screen saying I was running low on memory with the option to force quit one of a set of running application. Before I could respond by killing one of the Python programs, the system froze, hung completely and after about 5 minutes rebooted. No problem I thought, since I keep almost all stuff on some form of cloud storage (iCloud, GitHub, S3). When the system came back I discovered that Postgresql 15 was not running. After a lot of digging around I found that the following happened:

• The /usr/local/var folder was gone. This is the directory that contains the local Postgresql database I am running.
• Homebrew lost its permissions on the /opt folder.
• All of the terminal history of live sessions did not get written in the zsh history (this I rely extensively on).

I had "downgraded" from an M1 MPB 14 (64/10/1TB) to this machine when I left my last job. This is the first time that an out of memory error has caused some damage and it is actually the first time I have lost any files on a Mac (been exclusively using Macs since 2011). I usually run a slew of applications that are memory intensive (PyCharm, Datagrip, Safari, Firefox, Python programs) and had not rebooted the machine for several months. The problem may have been exacerbated by the huge amount of log data shown on all open terminal sessions.

Nothing was permanently lost and I got everything up and going after about two hours except for the database which I am now in the process of rebuilding from the original source data. But I will definitely go back to 64GB when I buy my next MBP (M4 probably) and will also make sure to reboot more frequently.

 

[frou]

I also had a system crash lately that seemed to partially corrupt the filesystem (some directories disappeared including in /usr/local). According to Disk Utility's First Aid feature, there are some uncorrectable anomalies in the filesystem, so I'm going to have to clean install the OS to be able to trust this machine again.

 

[splifingate]

!

*note to Self: don't forget to run SuperDuper! tmrw*...

 

[t2jd1967]

I also had a system crash lately that seemed to partially corrupt the filesystem (some directories disappeared including in /usr/local). According to Disk Utility's First Aid feature, there are some uncorrectable anomalies in the filesystem, so I'm going to have to clean install the OS to be able to trust this machine again.

Yeah, I was rather wanting to avoid a complete reinstall, so will see how it goes in the next couple of weeks. I really like APFS, but this is rather disconcerting to be honest. Why a crash can corrupt a file system is a mystery to me and it indicates a rather poor implementation. Oh well, we just going to have to live with it I guess.

 

[t2jd1967]

!

*note to Self: don't forget to run SuperDuper! tmrw*...

A number of years ago I was working off two systems and then decided to try to get as much stuff as possible in some form of cloud storage. Even though I have now gone back to a single system, I have kept this way of working going. The one thing that I find troubling is that it is unclear what would happen if the local iCloud drive gets corrupted, which hopefully would not result in erasing files on the online copy. I recently ditched Dropbox for Business because it simply was too expensive, but at least Dropbox had recovery.

Note to self: regularly back up the iCloud drive to S3 Glacier Deep Archive!

P.s. I love S3 GDA for long-term storage of video recordings, too bad retrieval is soo expensive!

 

[kagharaht]

I have a brand new iMac M3, 24gb, 2TB, 8/10, 4port. I did a new set up and no migration of anything. iCloud has all my stuff and one mail account on iCloud. After 14.4 update Mail started to quit on its own. Then in March 11/12 I was slapping some mov files together using Quicktime. I usually also have Mail-Safari-iMessage running in the background. Quicktime completely froze, turned my screen PINK and the iMac Shut down and rebooted. It then wanted me to verify the keyboard I'm using, which is what came withe the iMac, then the media keys just beeps wouldn't work. So I unpaired and plugged it back in with Cable and that fixed it. After that Mail just started quitting on its own again, also any DVD/BR I load doesn't show on the desktop, just the name.... A few days ago I finally had to do an Erase to Factory settings and start all over again. Again no restore from TM just sync iCloud. After the restore and set up, I couldn't modify the Date and Time on the menu bar. So I did a standard reboot and then it fixed that. I think Sonoma is not very stable. At least in my experience so far with a new iMac M3.

Oh and I still get the occasional external SSD just randomly disconnecting when used.

 

[splifingate]

A number of years ago I was working off two systems and then decided to try to get as much stuff as possible in some form of cloud storage. Even though I have now gone back to a single system, I have kept this way of working going. The one thing that I find troubling is that it is unclear what would happen if the local iCloud drive gets corrupted, which hopefully would not result in erasing files on the online copy. I recently ditched Dropbox for Business because it simply was too expensive, but at least Dropbox had recovery.

Note to self: regularly back up the iCloud drive to S3 Glacier Deep Archive!

P.s. I love S3 GDA for long-term storage of video recordings, too bad retrieval is soo expensive!

Yes; I use B2 . . . never had to retrieve anything, so-far, but--as GBS once said--"The optimist created the aeroplane, and the pessimist created the parachute."

All that being said, Sonoma (latest) on my M2 Mac Studio has been sublime...no data-loss; no dis-connects; no problems.

Nevertheless, I'm still packing a parachute

 

[Sumo999]

It's my understanding that pink screen means a kernel panic, though knowing that in itself may not be particularly helpful. When my M1 iMac was new peripherals and adapters were especially problematic in this regard.

In general, I do a periodic Safe Boot for preventative maintenance.

 

[t2jd1967]

I have a brand new iMac M3, 24gb, 2TB, 8/10, 4port. I did a new set up and no migration of anything. iCloud has all my stuff and one mail account on iCloud. After 14.4 update Mail started to quit on its own. Then in March 11/12 I was slapping some mov files together using Quicktime. I usually also have Mail-Safari-iMessage running in the background. Quicktime completely froze, turned my screen PINK and the iMac Shut down and rebooted. It then wanted me to verify the keyboard I'm using, which is what came withe the iMac, then the media keys just beeps wouldn't work. So I unpaired and plugged it back in with Cable and that fixed it. After that Mail just started quitting on its own again, also any DVD/BR I load doesn't show on the desktop, just the name.... A few days ago I finally had to do an Erase to Factory settings and start all over again. Again no restore from TM just sync iCloud. After the restore and set up, I couldn't modify the Date and Time on the menu bar. So I did a standard reboot and then it fixed that. I think Sonoma is not very stable. At least in my experience so far with a new iMac M3.

Oh and I still get the occasional external SSD just randomly disconnecting when used.

Interesting on the pink screen, that happened to me briefly as well before the reboot kicked in. I think the system running out of resources should not be possible, but apparently that safeguard is missing in macOS. I have never trusted the Time Machine for a system restore and the occasion that I did use it was for restoring user data only. It is weird that a clean install would throw up new problems. Did you create a new boot USB stick for a complete wipe of the M2 drive instead of using the built-in recovery? In my experience that works best and you can avoid most of the post install updates,

 

[t2jd1967]

Yes; I use B2 . . . never had to retrieve anything, so-far, but--as GBS once said--"The optimist created the aeroplane, and the pessimist created the parachute."

All that being said, Sonoma (latest) on my M2 Mac Studio has been sublime...no data-loss; no dis-connects; no problems.

Nevertheless, I'm still packing a parachute

I never used B2 and because I have some AWS experience, I am kind of defaulting to that platform. Worked on AWS Lambda for a while and that is where AWS really shines. And I have transferred my personal domain and email to AWS as well and it helps that everything is now together.

 

[t2jd1967]

It's my understanding that pink screen means a kernel panic, though knowing that in itself may not be particularly helpful. When my M1 iMac was new peripherals and adapters were especially problematic in this regard.

In general, I do a periodic Safe Boot for preventative maintenance.

Interesting, didn't know that a pink screen is equivalent to a BSOD. Well, you learn something everyday, not that I was looking forward to experiencing this one . Out of curiosity, what does a Safe Boot accomplish over a periodic regular reboot (which to be fair I tend to delay forever at my own peril).

 

[kagharaht]

Interesting on the pink screen, that happened to me briefly as well before the reboot kicked in. I think the system running out of resources should not be possible, but apparently that safeguard is missing in macOS. I have never trusted the Time Machine for a system restore and the occasion that I did use it was for restoring user data only. It is weird that a clean install would throw up new problems. Did you create a new boot USB stick for a complete wipe of the M2 drive instead of using the built-in recovery? In my experience that works best and you can avoid most of the post install updates,

I thought we no longer need to do that with iMac M3 with Sonoma.

Erase your Mac and reset it to factory settings - Apple Support

Use Erase All Content and Settings to quickly and securely erase all of your settings, data, and apps, while maintaining the operating system currently installed.

support.apple.com

Bootable USB

Create a bootable installer for macOS - Apple Support

You can use a USB flash drive or other secondary volume as a startup disk from which to install the Mac operating system. These steps are primarily for system administrators and other experienced users who are familiar with entering commands in Terminal.

support.apple.com

 

[Sumo999]

I believe that safe boot blocks some extensions and 3rd party stuff from loading, briefly checks the disk, deletes some caches, and should thus help isolate software from hardware issues.

With the sealed system volume reinstalling the OS is now redundant since it runs from a snapshot image by design.

 

[HDFan]

Why a crash can corrupt a file system is a mystery to me

If the system crashes while updating directories/files/pointers, without finishing the file system can be corrupted.

what does a Safe Boot accomplish over a periodic regular reboot (which to be fair I tend to delay forever at my own peril).

“ Verifies your startup disk and attempts to repair directory issues, if needed
Loads only required system extensions
Prevents login items from opening automatically
Disables user-installed fonts
Deletes font caches, the kernel cache, and other system cache files

”

Excerpt From
macOS Support Essentials 12 - Apple Pro Training Series
Benjamin G. Levy

‎macOS Support Essentials 12 - Apple Pro Training Series

‎Computers & Internet · 2022

books.apple.com

This material may be protected by copyright.

 

[splifingate]

I never used B2 and because I have some AWS experience, I am kind of defaulting to that platform. Worked on AWS Lambda for a while and that is where AWS really shines. And I have transferred my personal domain and email to AWS as well and it helps that everything is now together.

All Good if it works, Man

I guess your "too bad retrieval is soo expensive!" comment is what prompted me to reference B2.

 

[t2jd1967]

I thought we no longer need to do that with iMac M3 with Sonoma.

Erase your Mac and reset it to factory settings - Apple Support

Use Erase All Content and Settings to quickly and securely erase all of your settings, data, and apps, while maintaining the operating system currently installed.

support.apple.com

Bootable USB

Create a bootable installer for macOS - Apple Support

You can use a USB flash drive or other secondary volume as a startup disk from which to install the Mac operating system. These steps are primarily for system administrators and other experienced users who are familiar with entering commands in Terminal.

support.apple.com

Call me old fashioned, I like the ability to completely erase the volume so nothing gets left behind. You are right of course that it should not be necessary these days, unless you like a case sensitive file system. I decided not to do that when I got this machine and now I am regretting that. So am probably going to kill two birds with one stone, reset everything and get a case sensitive file system to boot. *)

*) Yes, Apple warns against this, which is kind of silly, and I think Apple should make this the default for the future.

 

[t2jd1967]

I believe that safe boot blocks some extensions and 3rd party stuff from loading, briefly checks the disk, deletes some caches, and should thus help isolate software from hardware issues.

With the sealed system volume reinstalling the OS is now redundant since it runs from a snapshot image by design.

I wouldn't say that it is redundant if you are getting loads of errors. But I take it this is not what you mean.

 

[t2jd1967]

If the system crashes while updating directories/files/pointers, without finishing the file system can be corrupted.




“ Verifies your startup disk and attempts to repair directory issues, if needed
Loads only required system extensions
Prevents login items from opening automatically
Disables user-installed fonts
Deletes font caches, the kernel cache, and other system cache files

”

Excerpt From
macOS Support Essentials 12 - Apple Pro Training Series
Benjamin G. Levy

‎macOS Support Essentials 12 - Apple Pro Training Series

‎Computers & Internet · 2022

books.apple.com

This material may be protected by copyright.

The directory was already gone after the system rebooted off the pink screen. But good to know for the future. It will probably take some good timing (i.e. power off) to make sure you first run a safe boot before the restart kicks in.

I am still at a loss while a kernel panic would lose a whole folder with loads of subfolders and files in. It should not happen and I am convinced that Apple made mistakes in its implementation of APFS, probably sacrificing integrity for speed. I am not saying that an open file cannot be corrupted, of course it can. But the result should never be loosing a bunch of stuff.

 

[t2jd1967]

All Good if it works, Man

I guess your "too bad retrieval is soo expensive!" comment is what prompted me to reference B2.

I record 4K videos (~35GB each) that I am probably only going to need occasionally in the future. The S3 Glacier Deep Archive in region US East is probably the cheapest storage around at $0.00099 per GB per month (~ $12/TB/year).

P.s. I think to some extend you can work around the retrieval costs by temporary spinning up an EC2 instance and take it from there. More hassle, but probably worth it next time around.

 

[frou]

If the system crashes while updating directories/files/pointers, without finishing the file system can be corrupted.

I'm no filesystem expert but isn't that like a '90s outlook? You'd think a modern well-designed filesystem would use all sorts of transactional/atomic techniques to be robust to crashing or power-loss.

 

[t2jd1967]

I'm no filesystem expert but isn't that like a '90s outlook? You'd think a modern well-designed filesystem would use all sorts of transactional/atomic techniques to be robust to crashing or power-loss.

I have to agree. Files getting corrupted, whilst not pleasant, is something I can understand and live with. Losing part of the folder structure should simply not be possible unless the carrier itself is corrupted.

 

[HDFan]

am still at a loss while a kernel panic would lose a whole folder

A folder in Unix is actually a file which has things like the file name, pointers to the file, etc. If that file gets corrupted and unusable in some way you lose the entire folder. Rarely happens but it is possible.

 

[t2jd1967]

A folder in Unix is actually a file which has things like the file name, pointers to the file, etc. If that file gets corrupted and unusable in some way you lose the entire folder. Rarely happens but it is possible.

I think it very much depends on the implementation whether folder corruption during write leads to data loss, and in my view Apple has prioritised speed over integrity. There is always a trade off to be made and here I think that Apple has unnecessarily thrown caution into the wind. Remember this is not some deeply nested tree folder that was lost, but the higher level folder /usr/local/var. That folder itself will not be modified much, because the Postgresql files are in folder further down called /usr/local/var/postgres. It also does not explain how permissions could be lost on the rather static folder /opt. All in all not great,