SSD/Flash storage in Macs: Premium prices for garbage?!

[k27]

Hector Martin works on Asahi Linux for AppleSilicon Macs:

"Well, this is unfortunate. It turns out Apple's custom NVMe drives are amazingly fast - if you don't care about data integrity. If you do, they drop down to HDD performance. Thread."

"For a while, we've noticed that random write performance with fsync (data integrity) on Asahi Linux (and also on Linux on T2 Macs) was terrible. As in 46 IOPS terrible. That's slower than many modern HDDs. We thought we were missing something, since this didn't happen on macOS."

There is more:

https://twitter.com/i/web/status/1494213855387734019

more privacy friendly link:

Hector Martin (@marcan@treehouse.systems) (@marcan42)

For a while, we've noticed that random write performance with fsync (data integrity) on Asahi Linux (and also on Linux on T2 Macs) was terrible. As in 46 IOPS terrible. That's slower than many modern HDDs. We thought we were missing something, since this didn't happen on macOS.

nitter.kavin.rocks

 

[leman]

The performance drop of F_FULLSYNC is indeed both surprising and concerning. I surely hope that this is due to lack of optimizations in the SSD controller and will be fixed with a software update.

There is also a secondary concern which is the leaky abstraction behind the new implementation. The awkward thing about fsync() is that it has never been never reliable - disk controllers often cheat and don’t do a full flush. An OS can’t do anything about it as the implementation details are hidden in the controller. So Apples overall approach - to provide an even stricter flush operation and ensure that it actually gets carried out - is sane (that’s one reason why Apple has been shipping drives with custom firmware for over a decade now). However, Apples new drives have such huge caches and flush so lazily that fsync practically becomes a no op. That is not how these things are supposed to work.

 

[k27]

"Chances are this happened because this design came from iDevices, where there's always a battery, so software will never hit drive cache related consistency/durability issues, so probably ~no iOS software ever uses a full sync, so nobody noticed it's slow."

Hector Martin (@marcan@treehouse.systems) (@marcan42)

Chances are this happened because this design came from iDevices, where there's always a battery, so software will never hit drive cache related consistency/durability issues, so probably ~no iOS software ever uses a full sync, so nobody noticed it's slow.

nitter.snopyta.org

 

[crazy dave]

The performance drop of F_FULLSYNC is indeed both surprising and concerning. I surely hope that this is due to lack of optimizations in the SSD controller and will be fixed with a software update.

There is also a secondary concern which is the leaky abstraction behind the new implementation. The awkward thing about fsync() is that it has never been never reliable - disk controllers often cheat and don’t do a full flush. An OS can’t do anything about it as the implementation details are hidden in the controller. So Apples overall approach - to provide an even stricter flush operation and ensure that it actually gets carried out - is sane (that’s one reason why Apple has been shipping drives with custom firmware for over a decade now). However, Apples new drives have such huge caches and flush so lazily that fsync practically becomes a no op. That is not how these things are supposed to work.

Hector's latest is that it is indeed a bug in the SSD controller firmware and could be fixed in a update.

 

[leman]

"Chances are this happened because this design came from iDevices, where there's always a battery, so software will never hit drive cache related consistency/durability issues, so probably ~no iOS software ever uses a full sync, so nobody noticed it's slow."

If your database software does not use full sync, then it can't claim that it is ACID. Database corruption is no joke. Indeed, a quick search of sqlite3 source code shows that they indeed use F_FULLFSYNC — as they should.

The reason why nobody noticed that it is slow is likely because these operations are extremely rare and are unlikely to impact performance. If you are constantly doing full storage flushes, you likely have a bigger issue in your software.

 

[jdb8167]

It isn't clear that this is really a problem for typical Mac hardware and macOS usage. It is a problem for Linux which is used in wider variety of situations than macOS. Linux is very common for enterprise and server deployments. Those cases are where this could be a real problem where very slow fsync writes are not acceptable but data loss on transactional writes is even worse. It sounds like Marcan has a reasonable solution for now which is a F_FULLSYNC timeout setting on fsync. Since it is unlikely anyone is ever going to use Asahi Linux in an environment where this will matter, I don't think this warrants much more attention. On the other hand, if it is a software or firmware bug, of course Apple should fix it.

Edit: BTW, there is also some discussion on Hacker News and it hasn't devolved into the normal "hate everything Apple" yet.

 

[crazy dave]

It isn't clear that this is really a problem for typical Mac hardware and macOS usage. It is a problem for Linux which is used in wider variety of situations than macOS. Linux is very common for enterprise and server deployments. Those cases are where this could be a real problem where very slow fsync writes are not acceptable but data loss on transactional writes is even worse. It sounds like Marcan has a reasonable solution for now which is a F_FULLSYNC timeout setting on fsync. Since it is unlikely anyone is ever going to use Asahi Linux in an environment where this will matter, I don't think this warrants much more attention. On the other hand, if it is a software or firmware bug, of course Apple should fix it.

The slowness of F_FULLSYNC does seem to be a (performance) bug. Maybe doing unnecessary actions or unnecessarily slow actions on flushes:

https://twitter.com/i/web/status/1494255649026625537

 

[leman]

It isn't clear that this is really a problem for typical Mac hardware and macOS usage. It is a problem for Linux which is used in wider variety of situations than macOS. Linux is very common for enterprise and server deployments. Those cases are where this could be a real problem where very slow fsync writes are not acceptable but data loss on transactional writes is even worse. It sounds like Marcan has a reasonable solution for now which is a F_FULLSYNC timeout setting on fsync. Since it is unlikely anyone is ever going to use Asahi Linux in an environment where this will matter, I don't think this warrants much more attention. On the other hand, if it is a software or firmware bug, of course Apple should fix it.

Edit: BTW, there is also some discussion on Hacker News and it hasn't devolved into the normal "hate everything Apple" yet.

I have read through the hacker news thread (a lot of excellent information there!) but it is still not quite clear to me why it is certain that Linux does a full flush on fsync(). How does one ensure that the drive controller does not cheat? That is not something the OS can affect right? As mentioned before, Apple has been shipping custom firmware on HDDs and SSDs for a long while now exactly to ensure that no cheating happens during flush.

 

[jdb8167]

I have read through the hacker news thread (a lot of excellent information there!) but it is still not quite clear to me why it is certain that Linux does a full flush on fsync(). How does one ensure that the drive controller does not cheat? That is not something the OS can affect right? As mentioned before, Apple has been shipping custom firmware on HDDs and SSDs for a long while now exactly to ensure that no cheating happens during flush.

This post by @marcan_42 (Hector Martin) is a very interesting clue.

 

[jdb8167]

I have read through the hacker news thread (a lot of excellent information there!) but it is still not quite clear to me why it is certain that Linux does a full flush on fsync(). How does one ensure that the drive controller does not cheat? That is not something the OS can affect right? As mentioned before, Apple has been shipping custom firmware on HDDs and SSDs for a long while now exactly to ensure that no cheating happens during flush.

If you are running a transactional database or other transactional service, you buy enterprise SSDs and HDDs. Those come with specs that guarantee full flush writes. Just like the custom firmware that Apple previously required for internal drives.

 

[leman]

If you are running a transactional database or other transactional service, you buy enterprise SSDs and HDDs. Those come with specs that guarantee full flush writes. Just like the custom firmware that Apple previously required for internal drives.

There are plenty of apps that use sqlite3 for document storage. I regularly use it myself. Or the types of databases such as git. It would be really unfortunate if you were to lose all your data to a corrupted database because of a sudden power loss.

 

[crazy dave]

Thankfully kernel panics alone shouldn't result in a failure.

https://twitter.com/i/web/status/1494294527523590145

I have read through the hacker news thread (a lot of excellent information there!) but it is still not quite clear to me why it is certain that Linux does a full flush on fsync(). How does one ensure that the drive controller does not cheat? That is not something the OS can affect right? As mentioned before, Apple has been shipping custom firmware on HDDs and SSDs for a long while now exactly to ensure that no cheating happens during flush.

Not sure either, but Hector seems convinced that is the behavior in Linux?

https://twitter.com/i/web/status/1494339221301383171

 

[leman]

Not sure either, but Hector seems convinced that is the behavior in Linux?

https://twitter.com/i/web/status/1494339221301383171

Maybe what he means is that Linux will send the full flush instruction to the drive controller on fsync(), while some other UNIX-like systems (like macOS or FreeBSD) can have a different behaviour. And I completely agree with him that this kind of system-dependent behaviour in a key functional area is problematic (but that seems to be a problem in the POSIX standard).

What confuses me though is how Linux is supposed to ensure that the drive controller does what Linux tells it to do. A drive can ignore or optimise the flush for all we know. From my limited understanding, full flush can be only ensured if the controller firmware is also open source and tested to do the right thing.

 

[jdb8167]

There are plenty of apps that use sqlite3 for document storage. I regularly use it myself. Or the types of databases such as git. It would be really unfortunate if you were to lose all your data to a corrupted database because of a sudden power loss.

You said that sqlite3 does the right thing already. It uses F_FULLSYNC as needed. Marcan isn’t saying fullsync doesn’t work, just that it is 10x slower than it should be. The problem is that normal fsync doesn’t periodically force fullsync on macOS as it does on Linux.

I have a UPS on my Mac and RAID that I use as a server because the RAID uses write caching and can corrupt on a power failure. Even though the Mac is Intel you aren’t guaranteed non-corruption on a power failure. On my M1 MacBook Air this is obviously not a problem.

 

[Slartibart]

can anyone of the more knowledgeable persons present make an educated guess wether this is the origin or cause for the various problems iOS/iPadOS devices have when writing to directly connected external media (with whatever file system).

 

[Gnattu]

wether this is the origin or cause for the various problems iOS/iPadOS devices have when writing to directly connected external media

No it is not. This only affects internal storage, and it is not even a big problem for battery-backed devices (it is too rare to run out of battery before the flush is completed). This WILL be a problem for desktop Macs, because they are too easy to lose power and subsequently lose data.

 

[k27]

I had to think of the ZFS developer's article about APFS. Some things about APFS are disappointing (especially if you use non-Apple drives - but the Apple drives... )

Adam Leventhal's blog » APFS in Detail: Data Integrity

dtrace.org

 

[jdb8167]

can anyone of the more knowledgeable persons present make an educated guess wether this is the origin or cause for the various problems iOS/iPadOS devices have when writing to directly connected external media (with whatever file system).

It seems likely. Maybe the reason third-party software seems not to have the same corruption issues is that they issue the F_FULLSYNC periodically.

 

[leman]

This WILL be a problem for desktop Macs, because they are too easy to lose power and subsequently lose data.

Why would it be? If you use the API correctly (with the F_FULLSYNC), you get guaranteed flush to physical storage. That is how Macs has operated since, well, forever.

 

[jdb8167]

No it is not. This only affects internal storage, and it is not even a big problem for battery-backed devices (it is too rare to run out of battery before the flush is completed). This WILL be a problem for desktop Macs, because they are too easy to lose power and subsequently lose data.

I don’t think this correct given what I’ve read. The issue is that F_FULLSYNC is slow and the reason that slowness doesn’t affect macOS is that macOS doesn’t do F_FULLSYNC on fsync where Linux does. Not doing full sync doesn’t seem like it is just on internal drives but a policy of macOS itself.

 

[leman]

I had to think of the ZFS developer's article about APFS. Some things about APFS are disappointing (especially if you use non-Apple drives - but the Apple drives... )

Adam Leventhal's blog » APFS in Detail: Data Integrity

dtrace.org

APFS is fine for what it is designed to do. The biggest eyebrow raiser for me is probably the lack of checksumming and auto-healing on data, but then again, it’s not a file system for general consumption. If Apple can handle that stuff on a lower level, why add complexity to the file system? Instead, a system like ZFS that targets all kind of hardware needs these tools to compensate for flaws at the lower level.

 

[leman]

I don’t think this correct given what I’ve read. The issue is that F_FULLSYNC is slow and the reason that slowness doesn’t affect macOS is that macOS doesn’t do F_FULLSYNC on fsync where Linux does. Not doing full sync doesn’t seem like it is just on internal drives but a policy of macOS itself.

The reason why the slowness doesn’t affect macOS tat much is because there are only very few instances where one has to use full flush - typically for atomic database operations. And correctly written Mac software will use the full sync where needed - and if it doesn’t use it than it is not correctly written.

P.S. Found this article where they summarize things nicely

Modern file systems always include a way to request that the data be "on physical media" before the call returns thereby enabling databases and other applications that require stringent data properties (e.g. the fsync() or fdatasync() system calls). Per above, the gotcha is that "on physical media" can mean "when the data is in the storage unit controller cache data and still vulnerable" and not when it is actually on long-term physical media.

In fact, using F_FULLSYNC on Apple devices is the only mechanism I am aware off that guarantees full flush to the physical layer. And the only reason why Apple can guarantee it is because they control the firmware. Even back in the age of the HDDs they would have custom firmware on the drives that support this functionality (which incidentally was one of the main reasons why HDDs were not considered user-replaceable). I simply do not see how Linux can guarantee the same behavior in fsync() since Lonux kernel does not control the hardware layer.

 

[jdb8167]

The reason why the slowness doesn’t affect macOS tat much is because there are only very few instances where one has to use full flush - typically for atomic database operations. And correctly written Mac software will use the full sync where needed - and if it doesn’t use it than it is not correctly written.

P.S. Found this article where they summarize things nicely



In fact, using F_FULLSYNC on Apple devices is the only mechanism I am aware off that guarantees full flush to the physical layer. And the only reason why Apple can guarantee it is because they control the firmware. Even back in the age of the HDDs they would have custom firmware on the drives that support this functionality (which incidentally was one of the main reasons why HDDs were not considered user-replaceable). I simply do not see how Linux can guarantee the same behavior in fsync() since Lonux kernel does not control the hardware layer.

I think we are saying the same thing. On Linux, apparently the only API to flush to the physical media is fsync which as you note with modern consumer SSDs and HDs isn’t guaranteed. Apple, apparently acknowledging that consumer hardware cheats, provides a non-standard non-POSIX API. Basically even though on Apple silicon hardware, F_FULLSYNC is slow, it is still more correct than the Linux solution for many users of consumer hardware as long as the macOS software is written correctly.

 

[Gnattu]

Why would it be? If you use the API correctly (with the F_FULLSYNC), you get guaranteed flush to physical storage. That is how Macs has operated since, well, forever.

Because portable POSIX applications would assume fsync() does what F_FULLSYNC does.

 

[jdb8167]

Because portable POSIX applications would assume fsync() does what F_FULLSYNC does.

Except as noted above, a lot of consumer hardware ignores flush commands. Apple set out to fix that. Unfortunately, on Apple silicon their non-standard solution is extremely slow.