Oh, I mixed up two problems.
1. macOS(or the XNU Kernel)'s fsync() does not flush cache
2. F_FULLSYNC is slow on Apple's nvme controller
Obviously second one only affects internal drive if the software uses F_FULLSYNC.
Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
SSD/Flash storage in Macs: Premium prices for garbage?!
- Thread starter k27
- Start date
- Sort by reaction score
1. macOS(or the XNU Kernel)'s fsync() does not flush cache
2. F_FULLSYNC is slow on Apple's nvme controller
Obviously second one only affects internal drive if the software uses F_FULLSYNC.
So if iPadOS Files isn’t issuing the full flush command and is only doing fsync calls then it is entirely possible that this is what is causing corruption problems on external media. Though if the drive is ignoring the flush command anyway, it isn’t clear what Apple can do to fix the problem. But since there are reports that the corruption doesn’t happen with File Browser Pro, there must be a fix.
Except POSIX makes no such guarantee. In fact, relying on POSIX doing the right thing for these kind of low-level implementation details is incredibly naive, more naive even than using C++ standard library and expecting good performance. If you develop stuff for macOS or FreeBSD and need to flush to physical storage, you have to use F_FULLSYNC — this is how it always has been.
It does flush cache — the filesystem/kernel cache. It does not necessarily ensure that the data 100% lands on the physical storage. It's just like a regular POSIX fsync().
And this would (almost?) only happen when you unplug that external media ungracefully, although this could be quite usual. But if you are doing this you will be at a very high risk to lose your data on any computer, because the OS may not even have the chance to issue the (correct) flush command before the drive is disconnected.
Now we have a problem. When I was using that, I waited until the drive's led stops flashing. But not all drives have an activity led so there is no way to tell when it is "safe" to unplug the drive, could be annoying if the data is important.
Well, sure, but POSIX fsync() is not guaranteed to flush the device cache. That’s the point.
P.S. I got very curious about all this and I have dogged a bit more. If I understand it correctly now, Linux fsync() invokes the NVME flush operation (which is a low level guaranteed flush command) while Apple does not. That is probably what Hector means by saying that Linux does full flush by default. If that’s the case then I understand the problem better I think.
It appears that no one understands file systems or devices here.
fsync is not a guarantee the device is consistent. It’s only a guarantee that the device has been told it needs to be consistent. What happens inside the device is undefined. There is no guarantee a flush will do what you ask. This is why it’s pushed up a level to the filesystem.
Journaling file systems only guarantee consistency at one point in time not at the point of an fsync. Usually it’s before the point in time the fsync took place. Even if the device was issued fsync it doesn’t mean that the consistency point is then at all. The only way to get fully consistent systems is to do writes to two nodes and do a two phase commit. This is obviously not practical.
What we have is a complete best effort system.
As for data loss this means virtually nothing for the modern OS with low file system transaction churn and cloud sync’ed files.
This is completely blown out of proportion.
fsync is not a guarantee the device is consistent. It’s only a guarantee that the device has been told it needs to be consistent. What happens inside the device is undefined. There is no guarantee a flush will do what you ask. This is why it’s pushed up a level to the filesystem.
Journaling file systems only guarantee consistency at one point in time not at the point of an fsync. Usually it’s before the point in time the fsync took place. Even if the device was issued fsync it doesn’t mean that the consistency point is then at all. The only way to get fully consistent systems is to do writes to two nodes and do a two phase commit. This is obviously not practical.
What we have is a complete best effort system.
As for data loss this means virtually nothing for the modern OS with low file system transaction churn and cloud sync’ed files.
This is completely blown out of proportion.
Last edited:
Maybe they just sold contaminated crap?! And the slowness “Bug” is a workaround?!
macdailynews.com
Western Digital is an Apple supplier. Kioxia NAND chips are used in many Apple products, including the iPhone 13 family, the 14- and 16-inch MacBook Pro, and iPad Pro.
SSD prices could spike after contamination of 6.5 billion gigabytes of Western Digital NAND chips
Western Digital says it's lost at least 6.5 exabytes (6.5 billion gigabytes) of flash storage due to contamination issues at NAND production…
macdailynews.com
Western Digital is an Apple supplier. Kioxia NAND chips are used in many Apple products, including the iPhone 13 family, the 14- and 16-inch MacBook Pro, and iPad Pro.
Last edited:
Doesn’t that sound like a problem in Asahi Linux? Pretty clear to me, if you provided evidence this is a universal problem (not, oh, it’s not, macOS works fine so definitely asahi linux)Hector Martin works on Asahi Linux for AppleSilicon Macs:
"Well, this is unfortunate. It turns out Apple's custom NVMe drives are amazingly fast - if you don't care about data integrity. If you do, they drop down to HDD performance. Thread."
"For a while, we've noticed that random write performance with fsync (data integrity) on Asahi Linux (and also on Linux on T2 Macs) was terrible. As in 46 IOPS terrible. That's slower than many modern HDDs. We thought we were missing something, since this didn't happen on macOS."
There is more:
more privacy friendly link:
Hector Martin (@marcan@treehouse.systems) (@marcan42)
For a while, we've noticed that random write performance with fsync (data integrity) on Asahi Linux (and also on Linux on T2 Macs) was terrible. As in 46 IOPS terrible. That's slower than many modern HDDs. We thought we were missing something, since this didn't happen on macOS.nitter.kavin.rocks
That was pretty much all pulled before it entered market. Pretty clear problem is in asahi linuxMaybe they just sold contaminated crap?! And the slowness “Bug” is a workaround?!
SSD prices could spike after contamination of 6.5 billion gigabytes of Western Digital NAND chips
Western Digital says it's lost at least 6.5 exabytes (6.5 billion gigabytes) of flash storage due to contamination issues at NAND production…
Western Digital is an Apple supplier. Kioxia NAND chips are used in many Apple products, including the iPhone 13 family, the 14- and 16-inch MacBook Pro, and iPad Pro.
Well it really does look like a firmware problem but one that isn’t too consequential for macOS. Some unusual software might be slow but correctly written macOS software should be fine. This isn’t a functional bug but a performance problem for unusual work loads (unusual on the Mac).
This is all curious. I have no reason to disagree with Hector and at the same time I spoke to a couple of people who have been or are at Apple (via Twitter). Both are familiar with macOS filesystem and firmware and have been briefed by Dominic Giampaolo, and both are adamant that unless you know the firmware honours write requests, you don't know! That is to say, never trust them. They treat third party drives as adversaries.
In any case, just in case some haven't seen it, it's worth checking out
for a good summary.
And:
They work at Apple and are very well versed with drive firmware and file systems.
In any case, just in case some haven't seen it, it's worth checking out
for a good summary.
And:
They work at Apple and are very well versed with drive firmware and file systems.
Not really. The issue seems to be a firmware performance bug that causes the SSD to be slow on flushes - basically writes from the SSD caches to the SSD - much slower than it should be by an order of magnitude. This happens on macOS and Linux and is likely the firmware. A performance bug in the firmware can be anything from performing an unnecessary action during a flush or performing a necessary action using the worst possible algorithm and thus unnecessarily slowly.
The other issue is more ... a design difference between macOS and Linux where Hector prefers the Linux version. On Linux, they use a flush during fsync (standard file system syncing) and they assume the flush command during fsync will be honored and, in practice, for direct-attached NVMe drives, it (often? always?) is. However, that flush doesn't *have* to be honored by an SSD controller. So Apple decided then to make an fsync that basically never flushes and provides a separate command that forces flushes to happen at specific times. Again, unfortunately this flush seems to be unusually slow, and is, again almost certainly a firmware bug as Hector can see no reason that a flush should be so slow.
Basically don't lose power on your desktop Mac if losing the last few seconds of writes to your internal SSD is crucial.
I've lost data in the past from being an unprepared idiot, but now everything is on a backup battery power supply.Basically don't lose power on your desktop Mac if losing the last few seconds of writes to your internal SSD is crucial.
A very good idea. I’ve been meaning to get a backup battery myself.
Does todays Eclectic Light aricle throw any light:
https://eclecticlight.co/2022/02/18/how-can-you-trust-a-disk-to-write-data/
https://eclecticlight.co/2022/02/18/how-can-you-trust-a-disk-to-write-data/
I remember that the developers of FileBrowser said they try to check this. In general FileBrowser reduces file corruption, file system inconsistency after unplugging is still a thing on iPadOS.