ssd swap - high usage of Terabytes Written

[Mike Boreham]

I am pretty sure the anomaly occurred because, shortly before that screenshot I had had 12 TB of prime data and backup drives connected for some time time. This would have caused the big mds process number. While they were connected the Total written number would have been much larger than 178, from all the backup data movement. When the screenshot was taken the drives had been disconnected, so the total written came right down, but I think the mds did not. The tests I mentioned above to reproduce this will confirm (or otherwise!).

This has not worked out as I expected. Contrary to previous results the Total data written did not change when the externals were disconnected.

Cancel that...I had forgotten to fully disconnect the externals instead of just unmounting them. The screenshots just before and after disconnecting the externals are below. The Total write number has come right down (from 533 to 71) but the Process totals have not changed....leading to the same apparently anomalous situation as before, that the Process numbers add up to more than the Total write number of 71.

 

[Maximara]

And nobody has to interpret the NVMe spec anyway because Apple provides an API for reading NVMe SMART data directly. Back some time ago I posted that it was up to the app developer to get the offsets of the data within the block of data returned by IOKit but that was wrong and based on old code in smartmontools meant for other SSDs and HDDs. The code that smartctl is using for reading the SMART data from the M1 is completely documented and supported within macOS Big Sur (and probably earlier). The library calls are straightforward and easy enough for anyone with experience with C or C++ to figure out. If the library functions aren't working correctly, that is fully on Apple for creating buggy software. But I don't believe that. From all my testing, the TBW values are working exactly as expected.

To sum up, Apple wrote the library function used to retrieve SMART data from the M1 SSD controller. They designed the SSD controller. They are clearly using the NVMe spec according to their source code header files. There is no ambiguity about this. I expect the usual suspects will simply ignore all this and continue saying that the data can't be relied on.

So the reason Fomalhaut via smartctl reporting 2% used with 70TB written reported didn't make sense was that was they where likely an old version of smartctl? It would explain the totally off the wall 6827 (170*100/2.49) TBW for the drive one gets if we use the worst possible number for that "2%". But the crosschecks sometime with the supposedly new code still produce bizarre and unlikely numbers.

 

[jdb8167]

So the reason Fomalhaut via smartctl reporting 2% used with 70TB written reported didn't make sense was that was they where likely an old version of smartctl? It would explain the totally off the wall 6827 (170*100/2.49) TBW for the drive one gets if we use the worst possible number for that "2%". But the crosschecks sometime with the supposedly new code still produce bizarre and unlikely numbers.

I'm seriously considering writing my own version of smartctl that only does one thing, read the internal SSD on a Big Sur Mac using Apple's "NVMeSMARTLibExternal.h". That way I could remove all the the cruft in smartctl that is meant for other operating systems and older drives. There is a lot and it is complicated so it is possible that there are bugs that others are seeing that I'm not.

 

[leons]

This has not worked out as I expected. Contrary to previous results the Total data written did not change when the externals were disconnected.

Cancel that...I had forgotten to fully disconnect the externals instead of just unmounting them. The screenshots just before and after disconnecting the externals are below. The Total write number has come right down (from 533 to 71) but the Process totals have not changed....leading to the same apparently anomalous situation as before, that the Process numbers add up to more than the Total write number of 71.

View attachment 1754163 View attachment 1754164

Great job and well-detailed info. Thanks for doing this!

 

[Maximara]

This has not worked out as I expected. Contrary to previous results the Total data written did not change when the externals were disconnected.

Cancel that...I had forgotten to fully disconnect the externals instead of just unmounting them. The screenshots just before and after disconnecting the externals are below. The Total write number has come right down (from 533 to 71) but the Process totals have not changed....leading to the same apparently anomalous situation as before, that the Process numbers add up to more than the Total write number of 71.

View attachment 1754163 View attachment 1754164

Well that does confirm something down right bizarre is going on. It seems to point to the "Process Name" section is in regards to all drives still connected or not while "Data written" only is in regards to still connected drives.

 

[Mike Boreham]

Well that does confirm something down right bizarre is going on. It seems to point to the "Process Name" section is in regards to all drives still connected or not while "Data written" only is in regards to still connected drives.

Yes, but whether it is bizarre perhaps depends what you want to use the data for. But the different treatment of the process values and the Disk total written is certainly inconsistent on the same display.

 

[jdb8167]

I'm seriously considering writing my own version of smartctl that only does one thing, read the internal SSD on a Big Sur Mac using Apple's "NVMeSMARTLibExternal.h". That way I could remove all the the cruft in smartctl that is meant for other operating systems and older drives. There is a lot and it is complicated so it is possible that there are bugs that others are seeing that I'm not.

Ok, I've got the basics working. I'll continue on it later but right now I'm getting the same DATA_UNITS_WRITTEN and HOST_WRITE_COMMANDS as smartctl which is encouraging. I'll post it to GitHub with a download when ready.

 

[haralds]

@leons You appear to be knowledgeable. I have combed this thread for specifics, but there are a lot of entries.
I am running my system with a number of apps always loaded. It's a MB Pro 16/2TB.
I seem to avg 150GB to 300GB written per day running 11.3b6. Should I worry?

 

[wirtandi]

Great job and well-detailed info. Thanks for doing this!

leons, you seem to be knowledgeable in this and I can see you have been very helpful, so appreciate your work.

My question is, could you summarize a list of things that I need to keep an eye on?

For example:

1. In the AM, I know I should go to the "Disk" tab and look at the "data written" down the bottom.
2. Download driveDX and see the SSD health indicator percentage and "data written".

But what else? I only know those two. What else should I keep an eye on?

 

[jasoncarle]

1 day, 1 hour, and 42 minutes uptime...

 

[leons]

@leons You appear to be knowledgeable. I have combed this thread for specifics, but there are a lot of entries.
I am running my system with a number of apps always loaded. It's a MB Pro 16/2TB.
I seem to avg 150GB to 300GB written per day running 11.3b6. Should I worry?

No. Don't worry and enjoy your wonderful machine. According to current data, your SSD should be good for 14 years+, assuming your HIGHEST level of data x365. There are some on this thread who feel the data is questionable. But even if you take 1/2 that number, you will be good for 7 years + at minimum. Enjoy!

 

[leons]

1 day, 1 hour, and 42 minutes uptime...

You are obviously Tab-Stingey, but your great-grandkids will enjoy their antique toy someday! ;-)

 

[leons]

leons, you seem to be knowledgeable in this and I can see you have been very helpful, so appreciate your work.

My question is, could you summarize a list of things that I need to keep an eye on?

For example:

1. In the AM, I know I should go to the "Disk" tab and look at the "data written" down the bottom.
2. Download driveDX and see the SSD health indicator percentage and "data written".

But what else? I only know those two. What else should I keep an eye on?

Sorry, I didn't answer your previous post because @Mike Boreham covered it well. Between what you list and what he said, that should be sufficient. Let us know approximately how much you are writing per day, and we can point you in the right direction. However, my guess is that you have no problem because I seem to remember you were getting a 512 SSD.

 

[Fomalhaut]

Can I ask what the heck some of you are doing with your computers that you have this much disk activity?

Here is my activity monitor currently on my M1 mini.

View attachment 1753703

Your disk usage is very low, which may be because you are few running applications (or no memory-hungry ones), probably don't have many browser tabs open, and aren't using much (or any ) swap. For work purposes, I need a minimum of 20 browser tabs, plus a number of applications (VSCode, Slack, Skype for Business / Zoom / Teams, Preview docs, often a Pages/Word/PPT/KeyNote document or two, DropBox, Google Backup, terminal windows....just stuff I need to do my job.

This generally means I'm using at least a couple of GB swap on my 16GB Mini, often quite a lot more. I've found that using swap is related to the volume of disk writes, and the solution seems to be just to control memory usage by removing caches and closing apps/windows. I'm generally writing 50-150GB per day. Realistically 32GB should be minimum for my usage, but they weren't offering this option for the M1 Macs, so I have to compromise either my workflow ("tidying up" more frequently) or accept reduced SSD life as the consequence.

 

[gank41]

@leons You appear to be knowledgeable. I have combed this thread for specifics, but there are a lot of entries.
I am running my system with a number of apps always loaded. It's a MB Pro 16/2TB.
I seem to avg 150GB to 300GB written per day running 11.3b6. Should I worry?

I have the same model also on 11.3b6, and am seeing the same numbers you’re seeing. We’re just fine.

 

[jdb8167]

I have a preliminary version of a command line tool that I called smartTBW. It does essentially the same thing as smartmontools (smartctl) and DriveDX but it is massively simplified. I wanted to be sure that no bugs crept in to verify that the existing tools are doing their jobs correctly.

I can verify that my results are identical to smartmontools. I kept everything as simple as possible so that anyone with the skills can review the code and see for themselves that Apple's libraries are doing all the heavy lifting. The only complication is dealing with 128-bit integers but that seems to be working.

You can look at the code here: https://github.com/jamesdbailey/smartTBW. I created a release to help with anyone who wants to install this without having to use Xcode. I don't know what kind of signing is required so if someone tries this and it doesn't work because of a signing error let me know and I'll look into it further.

The release is here: https://github.com/jamesdbailey/smartTBW/releases/download/v0.1/smartTBW

Edit: I'm not positive but it looks like you can either grant access to the tool explicitly in System Preferences->Security & Privacy->General after unlocking with an admin password or you can add Terminal to Developer Tools under System Preferences->Security & Privacy->Privacy. After unlocking click the + button and searching for Terminal, click the open button.

 

[qap]

I solved almost everything by disable the network "things/updates/check" when the Mac is in sleep, it also saves me a lot of battery, usually I "lost" 4/5% overnight (12/15h of sleep mode), with these tricks I have almost the same battery after I one day of sleep:

sudo pmset -a ttyskeepawake 0
sudo pmset -a tcpkeepalive 0

But this is usefuel only if you don't use your Mac often and you don't need the background updates, I usually open my MBA at work and close it after work, and I don't need it, or the updates, untill the next day. So for me it's perfect!

(PS: write 1 instead 0 to re-enable it)

 

[Mike Boreham]

Sorry, I didn't answer your previous post because @Mike Boreham covered it well. Between what you list and what he said, that should be sufficient. Let us know approximately how much you are writing per day, and we can point you in the right direction. However, my guess is that you have no problem because I seem to remember you were getting a 512 SSD.

An alternative (to Activity Monitor) way of checking the short term write rate (as opposed to the lifetime average rate) is with the popup in DriveDX:

It conveniently shows the delta write quantity, but doesn't unfortunately show the time interval that that applies over....which is actually the time since DriveDx was last launched. (The "last modified" date is the time of this launch not the previous). This is easily worked round and more convenient for me than disconnecting externals as required by Activity Monitor.
The quantity written in my screen shot is very low (14.2MB) because previous launch was only a few minutes earlier.

 

[osplo]

I have a preliminary version of a command line tool that I called smartTBW. It does essentially the same thing as smartmontools (smartctl) and DriveDX but it is massively simplified. I wanted to be sure that no bugs crept in to verify that the existing tools are doing their jobs correctly.

I can verify that my results are identical to smartmontools. I kept everything as simple as possible so that anyone with the skills can review the code and see for themselves that Apple's libraries are doing all the heavy lifting. The only complication is dealing with 128-bit integers but that seems to be working.

You can look at the code here: https://github.com/jamesdbailey/smartTBW. I created a release to help with anyone who wants to install this without having to use Xcode. I don't know what kind of signing is required so if someone tries this and it doesn't work because of a signing error let me know and I'll look into it further.

The release is here: https://github.com/jamesdbailey/smartTBW/releases/download/v0.1/smartTBW

Edit: I'm not positive but it looks like you can either grant access to the tool explicitly in System Preferences->Security & Privacy->General after unlocking with an admin password or you can add Terminal to Developer Tools under System Preferences->Security & Privacy->Privacy. After unlocking click the + button and searching for Terminal, click the open button.

AWESOME!!!!!

Thanks for taking the effort.

If my SSD can write 300 times its capacity then I am good for 25 years.

 

[Mike Boreham]

I have a preliminary version of a command line tool that I called smartTBW. It does essentially the same thing as smartmontools (smartctl) and DriveDX but it is massively simplified. I wanted to be sure that no bugs crept in to verify that the existing tools are doing their jobs correctly.

I can verify that my results are identical to smartmontools. I kept everything as simple as possible so that anyone with the skills can review the code and see for themselves that Apple's libraries are doing all the heavy lifting. The only complication is dealing with 128-bit integers but that seems to be working.

You can look at the code here: https://github.com/jamesdbailey/smartTBW. I created a release to help with anyone who wants to install this without having to use Xcode. I don't know what kind of signing is required so if someone tries this and it doesn't work because of a signing error let me know and I'll look into it further.

The release is here: https://github.com/jamesdbailey/smartTBW/releases/download/v0.1/smartTBW

Edit: I'm not positive but it looks like you can either grant access to the tool explicitly in System Preferences->Security & Privacy->General after unlocking with an admin password or you can add Terminal to Developer Tools under System Preferences->Security & Privacy->Privacy. After unlocking click the + button and searching for Terminal, click the open button.

Brilliant! and perfect for people like me for whom installing smartmontools and launching it with a terminal command is a bit of a challenge (although I have done it now). All I had to do with this is put it in the apps folder, launch it once with right click (to get round the unidentified developer) then in future launch it like another app. Thanks.

I did not do anything in Privacy Sys pref or anything else mentioned in your last paragraph.

 

[jdb8167]

Brilliant! and perfect for people like me for whom installing smartmontools and launching it with a terminal command is a bit of a challenge (although I have done it now). All I had to do with this is put it in the apps folder, launch it once with right click (to get round the unidentified developer) then in future launch it like another app. Thanks.

I did not do anything in Privacy Sys pref or anything else mentioned in your last paragraph.

I’m going to learn how to sign it so it passes Apple’s notarization checks. I have a dev account so it shouldn’t be too much trouble with a single file binary.

 

[Mike Boreham]

Edit: I'm not positive but it looks like you can either grant access to the tool explicitly in System Preferences->Security & Privacy->General after unlocking with an admin password or you can add Terminal to Developer Tools under System Preferences->Security & Privacy->Privacy. After unlocking click the + button and searching for Terminal, click the open button.

I’m going to learn how to sign it so it passes Apple’s notarization checks. I have a dev account so it shouldn’t be too much trouble with a single file binary.

Nothing in the last para (the Edit above) was needed at all here. All I did was download the exec file and put it in the Apps folder and launch it from there. I see that Terminal appears in my Privacy > Developer Tools but it is not checked so apparently not a necessary step. Nor is there anything in Full Disk Access or the General tab. The only one time step was the first right click to get round unidentified developer which I assume signing the app will avoid. Thanks again.

 

[ljg500]

Personally I have minimized the disk swapping issue by more controlled use of a machine not intended for work use- though I do intend to do more video and photo editing and well as DTP in the future. But the fact that I have this concern at all- is a disappointment- and despite the large number of posts here, I only suspect this is an Apple OS issue, perhaps aggravated by the M1 SOC architecture- and primarily impacting moderate to heavy users. Further, while the issue is mitigated by having 16 GB of ram- it is hardly eliminated- and for some users appears quite serious- even if the SSD TBW specs are not entirely clear for Apple's SSDs..

I realize this post is not particularly brilliant or helpful- but it tries to underscore the need for this issue to be taken seriously by Apple.

 

[Robospungo]

Say your machine’s SSD dies one day; are you able to boot using a USB-C external drive?

I’m on year 12 of my desktop. Only thing I ever had to replace was the hard drive.

 

[leons]

Personally I have minimized the disk swapping issue by more controlled use of a machine not intended for work use- though I do intend to do more video and photo editing and well as DTP in the future. But the fact that I have this concern at all- is a disappointment- and despite the large number of posts here, I only suspect this is an Apple OS issue, perhaps aggravated by the M1 SOC architecture- and primarily impacting moderate to heavy users. Further, while the issue is mitigated by having 16 GB of ram- it is hardly eliminated- and for some users appears quite serious- even if the SSD TBW specs are not entirely clear for Apple's SSDs..

I realize this post is not particularly brilliant or helpful- but it tries to underscore the need for this issue to be taken seriously by Apple.

As one who has been very vocal on this, I agree that Apple should take this seriously and solve the issue ASAP. In the meantime, I will continue to practice ways to mitigate the issue, while impacting workflow and usage patterns minimally. Interestingly enough, aside from considering a switch back to Safari from Firefox (not sure yet), I won't have to change anything after Apple implements their fix. If they never do (which I don't believe will happen), I personally wouldn't even notice.