I thought one could merely disable Spotlight with the following one line option?
sudo mdutil -a -i off
Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
ssd swap - high usage of Terabytes Written
- Thread starter Forti
- WikiPost WikiPost
- Start date
- Sort by reaction score
- Status
I thought one could merely disable Spotlight with the following one line option?
sudo mdutil -a -i off
I am now collecting data in a spreadsheet. I've had this mini for 66 days, and 148877.7759 GB have been written so far, averaging 2255.723877 GB per day.
From the data I have I calculated that the 100% lifetime would equal 2913401730 units written or 1488748.28 GB (1.488 PB), so if the trend continues as it's been these ~3 months, the 100% lifetime would be reached in 593 days, or October 18, 2022.
Unbelievable. I really hope it's just software that can be fixed.
So far, with Spotlight completely disabled, if it continues as it is it should be around 1 TB per day, down from the average of 2.25 TB that has been before. If it continues like this average, the 100% lifetime would be reached on October 27, 2024 instead, so another 2 years.
All just numbers... anyway I will get the system serviced before the AppleCare+ expires in December 2023 so that if I don't sell the computer at least I will have a new drive (actually whole computer perhaps because it's all integrated, right?).
From the data I have I calculated that the 100% lifetime would equal 2913401730 units written or 1488748.28 GB (1.488 PB), so if the trend continues as it's been these ~3 months, the 100% lifetime would be reached in 593 days, or October 18, 2022.
Unbelievable. I really hope it's just software that can be fixed.
So far, with Spotlight completely disabled, if it continues as it is it should be around 1 TB per day, down from the average of 2.25 TB that has been before. If it continues like this average, the 100% lifetime would be reached on October 27, 2024 instead, so another 2 years.
All just numbers... anyway I will get the system serviced before the AppleCare+ expires in December 2023 so that if I don't sell the computer at least I will have a new drive (actually whole computer perhaps because it's all integrated, right?).
Not sure, but before the mds process was running anyway and doing something with the external drives at least (because they are spinning disks and I can hear the noise). Since I did that change instead, the mds process is no longer running and the disks are very quiet apart from when backups happen.
I checked my M1 Mini this morning with "iostat -Id disk0", and the number had increased by over 1TB since yesterday evening, when I put the machine to sleep (c. 12 hours).
Code:
last night...
iostat -Id disk0
disk0
KB/t xfrs MB
112.13 32127102 3518007.07
this morning...
iostat -Id disk0
disk0
KB/t xfrs MB
127.47 39435330 4909130.73Most disk writes are from the kernel (just rebooted so can't screenshot, but it was about 2.6TB written with 4 days uptime)
I'm going over different theories why this might happen.
Sleep might be an effect that triggers that behavior but not the direct situation. Something, during sleep/hibernation etc triggers data to be dumped to the SSD when not enough memory is available and it does that a lot of times.
I can reproduce it during sleep but it's not the direct cause. Starting to think that apple does some M1 neural mumbo jumbo to pre-emptively cache memory stuff to the SSD and swap between active apps in memory. When the neural stuff go to sleep it. could trigger this mechanism - maybe the Mac thinks it needs to cache data and prepare for more to be loaded.
Spent the last few hours trying to reproduce this issue with a few apps and way under the threshold of memory consumed by a lot of apps and I'm not successful. But If I open up a lot of apps (14-15 GB of memory eaten) and I put it to sleep plugged in, bingo. So it's directly related to Swap being accessed. I can reproduce it 100% this way, but not when memory usage is low.
Edit: I ALSO HAVE TO HAVE THE MBPRO plugged in to have it happen. Does not happen when on battery power.
Edit 2: Could be that apple is logging power usage on battery during sleep, ends up eating a lot of ram after a while, so it begins swapping apps in ram. We need more people to test this out. And those that have the problem to try and reproduce it with way fewer apps.
Sleep might be an effect that triggers that behavior but not the direct situation. Something, during sleep/hibernation etc triggers data to be dumped to the SSD when not enough memory is available and it does that a lot of times.
I can reproduce it during sleep but it's not the direct cause. Starting to think that apple does some M1 neural mumbo jumbo to pre-emptively cache memory stuff to the SSD and swap between active apps in memory. When the neural stuff go to sleep it. could trigger this mechanism - maybe the Mac thinks it needs to cache data and prepare for more to be loaded.
Spent the last few hours trying to reproduce this issue with a few apps and way under the threshold of memory consumed by a lot of apps and I'm not successful. But If I open up a lot of apps (14-15 GB of memory eaten) and I put it to sleep plugged in, bingo. So it's directly related to Swap being accessed. I can reproduce it 100% this way, but not when memory usage is low.
Edit: I ALSO HAVE TO HAVE THE MBPRO plugged in to have it happen. Does not happen when on battery power.
Edit 2: Could be that apple is logging power usage on battery during sleep, ends up eating a lot of ram after a while, so it begins swapping apps in ram. We need more people to test this out. And those that have the problem to try and reproduce it with way fewer apps.
ME!
that and my devices took a step back recently last month were i realize is playing their "buy new" game.
another reason im not buying a new M1 is since i do not really need a new notebook
but the biggest reason is that most iPad apps are not available on the M1 yet or even never.
my macbook air from 2010 streamed a NHL game without a problem besides the fan going on 3 nites ago, so why do i need a new MacBook?
This aligns with the behaviour I see on my M1 Mini, which remains plugged in and (hopefully) sleeping. Memory is generally fully used with at least 5GB swap in use.
I’ll try to close apps down before sleeping to see if it makes any difference.
Just wanna say, many people like me appreciate your help and experiment in this. Please keep it up.
If only Apple would just release a statement.....would make things so much easier for everyone
Yeah where is Apple in this the ****? Like that issue has been around now for weeks officially and they don't respond?
Hopefully, they are aware of the issue and are investigating. It would be a very bad look for them to have large quantities of their first-generation Apple Silicon machines starting to fail in two years' time.
This may be a Big Sur issue though, so let's get more data-points from those users - who should more numerous than M1 owners if a large proportion of Intel Mac users update their OS.
Hello, I thought I'd share my experience with this so far. I am a photographer and I see this issue mainly when running Lightroom Classic. My Brave browser writes a bit here and there, but when Lightroom is launched, Kernel begins writing sometimes like 10GB or more almost instantly. Even running in the background it's writing 10's of GBs. It's written about 4TB's in the 4 days it's been on with relatively light use and not much Lightroom compared to normal. I've had this machine for 1 month and according to DriveDx I've written 92TB and used up 9% of the SSD lifespan.
I called Apple today to talk about this and feel I made a little progress, but overall the people I spoke to seem to ultimately feel it's not an issue even though they can't tell me why it's not. A bit frustrated and concerned. But again to reiterate, I can see these writes happening pretty much whenever I open Lightroom.
I called Apple today to talk about this and feel I made a little progress, but overall the people I spoke to seem to ultimately feel it's not an issue even though they can't tell me why it's not. A bit frustrated and concerned. But again to reiterate, I can see these writes happening pretty much whenever I open Lightroom.
Attachments
Are you using Lightroom Classic hardware acceleration? You can check by going into Preferences > Performance. Others in this forum have pointed out that they see high TBW activity with it on.
All I have is Auto, Custom, and Off. I have switched it off and will try it for a bit and see how that effects the SSD writes if at all.
Edit: After switching GPU Accel off, I am just browsing my Library and I have already seen about 20GB be written to SSD from Kernel task in the last few minutes.
Thanks for that. I would be pretty alarmed to see 9% of my estimated SSD lifespan used in one month. Do we know the figure is accurate? How is it being calculated?
I am also curious why Apple's impression was that "it's not a big deal". At the same rate of usage, your SSD would become unreliable in less than a year. I hope they meant "the numbers are not accurate", or "the SSD will still last for many years", rather than "it's not a big deal for us...you'll just buy a new machine when it fails...<ka-ching!>".
I'll be watching developments closely.
I won't be upgrading my MBP16 to Big Sur until it's been demonstrably stable for several more months or releases.
Does anyone know the TBW of the SSDs that Apple is using, for the various capacities?
While enterprise SSDs (probably SLC NAND) can have TBW figures in the tens of thousands (often with about 3 full drive writes per day for 5 years [DWPD = 3 x capacity]), the consumer-grade SSDs look much less robust.
This article: https://www.chillblast.com/learn/why-tbw-matters-in-ssds/ mentions a 1TB Samsung EVO 970 as having a TBW of 300TB.
If we're seeing M1 Macs having used half that figure in less than 3 months, there is a very serious issue if the numbers are correct.
The same article states "...at a 1TB capacity works out to a DWPD of over 164GB. Even that is a figure that very few users are likely to get even close to...". Well, I have seen 5-6x times this, per day, on my M1 Mini in the last few days. I am worried!
While enterprise SSDs (probably SLC NAND) can have TBW figures in the tens of thousands (often with about 3 full drive writes per day for 5 years [DWPD = 3 x capacity]), the consumer-grade SSDs look much less robust.
This article: https://www.chillblast.com/learn/why-tbw-matters-in-ssds/ mentions a 1TB Samsung EVO 970 as having a TBW of 300TB.
If we're seeing M1 Macs having used half that figure in less than 3 months, there is a very serious issue if the numbers are correct.
The same article states "...at a 1TB capacity works out to a DWPD of over 164GB. Even that is a figure that very few users are likely to get even close to...". Well, I have seen 5-6x times this, per day, on my M1 Mini in the last few days. I am worried!
I was seeing huge writes to the internal disk on my 16/256 M1 mini a couple of days ago - on the order of 1 TByte per day. Fortunately, I noticed this when the total writes had only reached about 4 TByte and have since managed to, temporarily at least, put a stop to it.
In short: for me it was Safari that caused trouble, possibly in combination with some Big Sur swapping issue.
Hopefully this will help someone else who is experiencing this issue. It may not be Safari in your case, but it might still be a good idea to take a closer look at what is being swapped out.
First, I verified over 24 hours that Activity Monitor was showing numbers that were within 10% of what smartctl reported, so there was no question about there being any kind of error with the interpretation of the SMART data.
By far, most of the writes were done by kernel_task, which was writing 10x+ more than it was reading.
Anyway, I noticed that swap use was very high, often at over 10 GByte. That in itself would not necessarily mean that a lot of writes needed to take place, especially not without corresponding reads. Nevertheless, I tried shutting down everything I had running (Xcode, GarageBand, Steam, Safari, Mail etc). With everything closed, my swap use dropped to zero, along with the writes. Bringing things up again brought the massive swap and writes back as well.
A little bit of testing showed that, at least in my case, it was Safari that was causing the main part of the swap use, and AFAICT nearly all of the writes by kernel_task. It should be noted that I am normally a _very_ heavy browser tab user - nearly always having hundreds of tabs open. In this particular case, I was at 528 tabs. With some tabs, for whatever reason, using over 200 MBytes according to Activity Monitor, that was clearly a very bad idea.
For now, I have "solved" the issue on my M1 by spending a couple of hours sorting and bookmarking hundreds of browser tabs. Only a dozen tabs open now but, with everything else back to about the same state as when I was seeing 1 TByte writes per day, the kernel_task write counter does not seem to move at all (0.5 GByte swap).
Of course, this is not the way I would like things to work, and I have never seen anything like this behaviour before. I'll be looking for a more resource friendly browser to use, and hoping for some kind of fixes to Big Sur and/or Safari.
Perhaps I'll end up spending the money I saved by not upgrading the SSD on the M1 to buy an external Thunderbolt SSD instead to run the OS from. ;-)
In short: for me it was Safari that caused trouble, possibly in combination with some Big Sur swapping issue.
Hopefully this will help someone else who is experiencing this issue. It may not be Safari in your case, but it might still be a good idea to take a closer look at what is being swapped out.
First, I verified over 24 hours that Activity Monitor was showing numbers that were within 10% of what smartctl reported, so there was no question about there being any kind of error with the interpretation of the SMART data.
By far, most of the writes were done by kernel_task, which was writing 10x+ more than it was reading.
Anyway, I noticed that swap use was very high, often at over 10 GByte. That in itself would not necessarily mean that a lot of writes needed to take place, especially not without corresponding reads. Nevertheless, I tried shutting down everything I had running (Xcode, GarageBand, Steam, Safari, Mail etc). With everything closed, my swap use dropped to zero, along with the writes. Bringing things up again brought the massive swap and writes back as well.
A little bit of testing showed that, at least in my case, it was Safari that was causing the main part of the swap use, and AFAICT nearly all of the writes by kernel_task. It should be noted that I am normally a _very_ heavy browser tab user - nearly always having hundreds of tabs open. In this particular case, I was at 528 tabs. With some tabs, for whatever reason, using over 200 MBytes according to Activity Monitor, that was clearly a very bad idea.
For now, I have "solved" the issue on my M1 by spending a couple of hours sorting and bookmarking hundreds of browser tabs. Only a dozen tabs open now but, with everything else back to about the same state as when I was seeing 1 TByte writes per day, the kernel_task write counter does not seem to move at all (0.5 GByte swap).
Of course, this is not the way I would like things to work, and I have never seen anything like this behaviour before. I'll be looking for a more resource friendly browser to use, and hoping for some kind of fixes to Big Sur and/or Safari.
Perhaps I'll end up spending the money I saved by not upgrading the SSD on the M1 to buy an external Thunderbolt SSD instead to run the OS from. ;-)
Is there some reason not to use the mdutil command instead?
sudo mdutil -i off /Volumes/<Data>I haven't needed to do this but it seems like disabling a system daemon is overkill when there is a standard way to do the same thing.
It comes from the drive itself. There is a standard for reading the SMART data from a NVMe SSD so I assume that Apple is using it correctly and there are multiple reports of people testing against Activity Monitor to say that it is right.
The 9% in this case implies that the TBW for that drive is about 1 PBW (1,000 TBW).
Nice report. Your usage is quite similar to mine (16/512GB M1 Mini; I also have lots of tabs open for work, but I don't think I've ever had over 500
)I also see a lot of Safari usage (after reboot about 4 hours ago):
This is with about 95 tabs open.
In my experience, 200MB for a browser tab is quite low. Many of them of over 1GB (unless you're referring to "Real Mem" and not just "Memory" (which is some combination of real and virtual memory I think)). Here are my top memory-hogs:
Note that Macforums is in 3rd place with 1.03GB - and only a single tab open!
I'm going to have a cleanup of my Safari tabs before sleeping the machine (a good thing to do in any case!), and see if that helps the overall usage.
My disk writes over the previous 3 days were about 2.6TB, including over 1TB during the night, so clearly something was going on. Let's see if the reboot and keeping memory usage lower (particularly Safari) will help matters.
Well, we do have some commonalities. Adobe Lightroom Classic is a memory hog (as reported above), and 528 tabs open in any browser are nice pigs too. Things for these users, at least, go back to normal when not using these memory abusers.
These two cases seems to point to the swapping management.
I've used mine since December 12, 2020. Reinstalled macOS a couple of times due to wanting to return it, but kept changing my mind. I also downloaded pretty big files (> 40 GB in size a couple of times). I think mine is pretty normal. I use Safari Technology Preview (121 as of right now) for my main browser with Chrome for when things are not behaving properly on Safari.
3.8 TB written according to DriveDX.
3.8 TB written according to DriveDX.