What's next after HFS+?

[2984839]

Leaving aside the features like better data integrity protection, I don't understand the mentioned problems with larger storage now and in the future when HFS+ is stated to support volumes and file sizes both up to 8 Exabyte and a maximum number of 4,294,967,295 files for each volume.

Does this just work in theory and are there much lower limits in reality?

Could anyone please explain the issues to a stupid person like me?

According to Apple, 8 EB is indeed the limit (https://support.apple.com/en-us/HT201711) so file size is not going to be an issue.

HFS+ has much more important shortcomings. It was designed for big-endian PowerPC processors, so metadata reads on the amd64 architecture requires byte-swapping to little endian. It uses case insensitive strings. It lacks checksumming for data integrity. It splits resource information (i.e., metadata) and data into separate forks, which complicates copying to and from non HFS+ filesystems, as the metadata ends up disappearing when the non HFS+ filesystem is unprepared to handle it. This additionally has implications for secure erasure, since the resource fork must also be overwritten along with the data.

Apple has tried to find ways around these problems and add features like journaling, but the result is a hacked together mess of a filesystem that really ought to be replaced.

 

[Markoth]

According to Apple, 8 EB is indeed the limit (https://support.apple.com/en-us/HT201711) so file size is not going to be an issue.

HFS+ has much more important shortcomings. It was designed for big-endian PowerPC processors, so metadata reads on the amd64 architecture requires byte-swapping to little endian. It uses case insensitive strings. It lacks checksumming for data integrity. It splits resource information (i.e., metadata) and data into separate forks, which complicates copying to and from non HFS+ filesystems, as the metadata ends up disappearing when the non HFS+ filesystem is unprepared to handle it. This additionally has implications for secure erasure, since the resource fork must also be overwritten along with the data.

Apple has tried to find ways around these problems and add features like journaling, but the result is a hacked together mess of a filesystem that really ought to be replaced.

Why don't you just use OpenZFS then? I've found it to be quite stable so far (at least as long as you don't try to use it with USB storage). Buy a couple hard drives and make a ZFS RAID of one kind or another and keep your important data on it. A native solution would be ideal, but Apple seems to already be working on that in the form of Core Storage. Once Apple has a use for extra features, I have no doubt they'll add them. For those of us that already have a use for those features, we have OpenZFS to tide us over until then.

 

[Erdbeertorte]

According to Apple, 8 EB is indeed the limit (https://support.apple.com/en-us/HT201711) so file size is not going to be an issue.

HFS+ has much more important shortcomings. It was designed for big-endian PowerPC processors, so metadata reads on the amd64 architecture requires byte-swapping to little endian. It uses case insensitive strings. It lacks checksumming for data integrity. It splits resource information (i.e., metadata) and data into separate forks, which complicates copying to and from non HFS+ filesystems, as the metadata ends up disappearing when the non HFS+ filesystem is unprepared to handle it. This additionally has implications for secure erasure, since the resource fork must also be overwritten along with the data.

Apple has tried to find ways around these problems and add features like journaling, but the result is a hacked together mess of a filesystem that really ought to be replaced.

Thank you for your answer.

You wrote it uses case insensitive strings, but I can choose case sensitive (also with encryption) when formatting any drive in Disk Utility.

Strange that according to Apple the maximum number of files is only half of what I read in Wikipedia...

 

[\-V-/]

Apple removed hardware RAID with El Capitan so you can only use software-based RAID now. Very frustrating.

 

[2984839]

Thank you for your answer.

You wrote it uses case insensitive strings, but I can choose case sensitive when formatting any drive in Disk Utility.

Yes, that's one of their hacks to deal with the fact that a lot of Unix software expects case sensitivity, since the entire Unix world except Apple uses a case-sensitive filesystem. Apple's use of case insensitivity by default has also resulted in even more workarounds by software developers to make sure their programs don't run into issues when faced with HFS+. It's all a big mess.

 

[Erdbeertorte]

Apple removed hardware RAID with El Capitan so you can only use software-based RAID now. Very frustrating.

But the RAID Utility is still there. I read somewhere here that you can only launch it when you have a Mac Pro and maybe special Apple RAID drives, but I am not sure...

 

[2984839]

Why don't you just use OpenZFS then? I've found it to be quite stable so far (at least as long as you don't try to use it with USB storage). Buy a couple hard drives and make a ZFS RAID of one kind or another and keep your important data on it. A native solution would be ideal, but Apple seems to already be working on that in the form of Core Storage. Once Apple has a use for extra features, I have no doubt they'll add them. For those of us that already have a use for those features, we have OpenZFS to tide us over until then.

Because I don't run OS X.

However, OpenZFS is probably a good starting point if Apple ever wanted to implement ZFS. I would still lean towards HAMMER2 over ZFS for simplicity's sake, though that isn't even ready on DragonFly yet.

 

[\-V-/]

But the RAID Utility is still there. I read somewhere here that you can only launch it when you have a Mac Pro and maybe special Apple RAID drives, but I am not sure...

View attachment 607801

View attachment 607802

I don't know what it's supported on, but none of the devices I was using it on (several) work anymore since El Capitan. It was a frustrating and silent removal and it makes me cringe:

People are pretty frustrated with this. Not just me.

 

[Erdbeertorte]

I don't know what it's supported on, but none of the devices I was using it on (several) work anymore since El Capitan. It was a frustrating and silent removal and it makes me cringe:

People are pretty frustrated with this. Not just me.

It's strange that it comes with a clean El Capitan Installation, when it is not supported anymore. I have even SIP disabled and can't launch it. Were you able to launch it on your Macs before El Capitan?

I never used any RAID and only heard about and saw myself that it isn't available in the new Disk Utility anymore, but should still work with "diskutil" in the Terminal. Is that just the Software RAID?

 

[Erdbeertorte]

I read somewhere here that you can only launch it when you have a Mac Pro and maybe special Apple RAID drives, but I am not sure...

@\-V-/

I found that post here:

https://forums.macrumors.com/threads/disk-utility-replacement.1933774/page-2#post-22187051

 

[\-V-/]

Yes, I have several RAID set ups on several computers that I had to back up and reconfigure because of this mess. RAID was fine before. And the example above is from a clean install of El Capitan. Everything has to be done in Terminal now I believe. -.-

You used to be able to set up RAID via Disk Utility but they removed that option and that's how I set up my original RAID configurations.

 

[Erdbeertorte]

You used to be able to set up RAID via Disk Utility but they removed that option and that's how I set up my original RAID configurations.

I have got a modified version of the old one that can be started und should be supposed to work in El Capitan. (Don't know where I got it from .)

Here is an Instruction but no download: https://justus.berlin/2015/10/restore-old-disk-utility-in-os-x-el-capitan/

 

[Jago]

Some of the comments here make me facepalm pretty hard since it's obvious that many posters do not have any actual real life experience with filesystems discussed.

1) ZFS CPU impact is miniscule (assuming you don't use dedup or compression). 5 years ago I used to run a NAS with a dual core Atom CPU without any problems.

2) ZFS RAM requirements are overblown. 8gb is more than plenty for basic use. In fact, my 5 year old NAS had just 4gb and even that was fine, as long as the pool wasn't larger than 2TB.

3) "HAMMER2 offers similar features with much less code and would be a better choice to port to OS X, both for simplicity and licensing reasons" - if Hammer code is both easier to work with as well as having better licensing, why isn't there even a single major network storage vendor using it in their products?

 

[2984839]

Some of the comments here make me facepalm pretty hard since it's obvious that many posters do not have any actual real life experience with filesystems discussed.

1) ZFS CPU impact is miniscule (assuming you don't use dedup or compression). 5 years ago I used to run a NAS with a dual core Atom CPU without any problems.

2) ZFS RAM requirements are overblown. 8gb is more than plenty for basic use. In fact, my 5 year old NAS had just 4gb and even that was fine, as long as the pool wasn't larger than 2TB.

3) "HAMMER2 offers similar features with much less code and would be a better choice to port to OS X, both for simplicity and licensing reasons" - if Hammer code is both easier to work with as well as having better licensing, why isn't there even a single major network storage vendor using it in their products?

HAMMER2 isn't production ready yet.

 

[Markoth]

HAMMER2 isn't production ready yet.

DragonFly BSD is a neat project to watch in general. Its approach to multitasking is also interesting.

 

[2984839]

DragonFly BSD is a neat project to watch in general. Its approach to multitasking is also interesting.

Yes, it is a very interesting OS. I have T420 with DF installed on an Ultrabay HDD and it is silky smooth. Not ready to replace OpenBSD as my main OS and may never will be, but that's more personal preference.

HAMMER (both the original and HAMMER2) offers quite a few features that would benefit OS X users and is simple to use. Some of the following commands are the ones I find most useful.

View history of a file:

hammer history /path/to/file

View history of a file as a unified diff:

undo -d /path/to/file

Restore a deleted file:

undo -o restoredfile.txt deletedfile.txt

Create a one time mirror of the directory "/home/joe/important-stuff" called "/important-stuff-backup":

hammer mirror-copy /home/joe/important-stuff /important-stuff-backup

Create a streaming mirror. Similar to above, but will continuously copy any changed or new files in "/home/joe/important-stuff" into "/important-stuff-backup":

hammer mirror-stream /home/joe/important-stuff /important-stuff-backup

 

[grahamperrin]

… I suggest there are only 4 paths:

…
3. Apple stays with HFS as-is; perhaps minor tweaks, but nothing to address 50TB+ local storage. They wait/expect third-party/NAS/Cloud/Something new to address the issue for large storage needs.
…

Upthere Home for OS X will not run on OS X 10.9.5, so I can't easily tell how it works. In the meantime, the (very) little that I have read about UpFS reminds me of a patent application by Apple. https://twitter.com/grahamperrin/status/683604357615583232

… The goal is to have the user NOT notice the filesystem. …

Smart.

I'll be interested to see how things develop around UpFS.

… obvious that many posters do not have any actual real life experience with filesystems discussed.

+1

1) ZFS CPU impact is miniscule (assuming you don't use dedup or compression). 5 years ago I used to run a NAS with a dual core Atom CPU without any problems.

2) ZFS RAM requirements are overblown. 8gb is more than plenty for basic use. In fact, my 5 year old NAS had just 4gb and even that was fine, as long as the pool wasn't larger than 2TB.
…

For a few months in 2015 I happily booted from ZFS with just 2 GB memory in a circa 2007 notebook. Eventually I realised that one of the two 2 GB modules was not properly seated. Now with 4 GB the experience is very pleasing.

… ZFS seems to gaining traction in the storage world …

Don Brady now works in the High Performance Data Division at Intel. Some of his ZFS-related work there is outlined in presentations that are linked from the OpenZFS site.

I don't think Apple ever intends to take mass local-storage seriously. …

https://twitter.com/DonJBrady/status/525142433705639936 I detect some irony.

It's a myth that file systems such as ZFS can only be useful for large storage solutions. …

… ZFS … I used it on a laptop for years running FreeBSD and was satisfied with the performance, and that was with 4 GB of RAM and an i5-540M. …

Ergo Vista 621 here, Intel Core2 Duo with PC-BSD 11.0-CURRENTDEC2015. 11.0-CURRENTJAN2016 is a very recent appearance; not yet announced.

… Mac … Lots of applications expect certain behaviour as it is implemented by HFS+. …

HFS Plus-specific coding; other weird and/or sloppy things that are done but not documented by Apple; and the problems are not limited to windowed applications. Spotlight- and other search-related technologies are, or were, particularly troublesome.

https://github.com/openzfsonosx/zfs/issues/278 was entertaining. Disbelief, closure, repetition, reopening, a quick fix and "… Apple DTS ticket #619066023 but I have had little success with DTS in the past. …".

Apple seems to be thinking that all data shall live in the cloud. …

See above, UpFS etc..

… can't see the value of using ZFS for small systems. …

Verifiable integrity of data, and more.

… Apple has no storage products, and none likely coming. …

Sites such as Patently Apple might hold some clues. Some of the more interesting patent applications gain little publicity.

… difficult to reduce functionality to get only the features you want. …

With a well-designed operating system it's really not difficult to get what's required from ZFS.

… ZFS is not for casual, non-tech users. …

Where the operating system makes appropriate use of the file system: ZFS is good for casual, non-tech users.

ZFS is outrageously complex. …

I'm never outraged by ZFS. Maybe because I had a good introduction to it.

Without a good introduction, any file system might seem unfriendly. I can throw all readers here in at the deep end with TN1150; someone can describe that as outrageously complex.

… A ZFS based NAS does not run on a low-end processor. … minimum is about $1K for an entry level built it yourself ZFS file server. …

Or £0.00 for a hand-me-down 2009 Dell Inspiron 545, and I stuck in a couple of hard disk drives that were discarded from the office.

you CAN set up a BSD/ZFS system on smaller hardware. I've done it. Performance is poor but at least you can learn about ZFS.

The Inspiron has been fine for backups to FreeNAS.

… to the average Mac user.

Apple can't sell a new filesystem as a new feature without first demonstrating, in layman's terms, what was wrong with the old one and why it was not possible to fix the problem until now.

… I've been using OpenZFS on OS X for awhile now, and it's working great! Just don't use it with a USB-attached drive …

Great for some use cases, but not good enough for me to recommend it without hesitation. OS X 10.9.5 stops responding with OpenZFS on OS X 1.4.5 … and after that, I ceased to follow the project (more broadly: I'm switching away from Apple products).

Incidentally, problems with USB 2.0 for storage are fairly generic. Not specific to ZFS.

… can't boot from ZFS, so the boot partition still has to be either HFS+ or Core Storage. …

With or without Core Storage, HFS Plus is required to boot OS X.

Within Apple's File System Programming Guide, File System Details (2015-03-09, highlights) states that OS X can boot from UFS, which may be true, but with modern versions of the OS it's no longer possible to install the OS to that file system. Within the Mac Technology Overview, the relevant part of Kernel and Device Drivers Layer (2015-10-21) is years out of date.

Some time in the past it was possible to boot Mac OS X from ZFS, but I never saw that; I expect the details to remain private. Instead, there's interest in more modern approaches e.g. PC-BSD and FreeBSD booting from ZFS.

… HFS+ … splits resource information (i.e., metadata) and data into separate forks …

There's widespread confusion around uses of resource forks. This topic is not the place to offer corrections, but here's a fun tidbit: AppleFSCompression may use a resource fork for a file that has no resource fork

… the entire Unix world except Apple uses a case-sensitive filesystem. …

I have a hunch that at least one such OS other than OS X uses case insensitivity by default for something …

 

[Markoth]

Great for some use cases, but not good enough for me to recommend it without hesitation. OS X 10.9.5 stops responding with OpenZFS on OS X 1.4.5 … and after that, I ceased to follow the project (more broadly: I'm switching away from Apple products).

Incidentally, problems with USB 2.0 for storage are fairly generic. Not specific to ZFS.

The cases of OS X not responding, at least for me, were 100% when I was using it with a USB-attached device of some sort, including the SD card slot (which is internally hooked up as a USB device). I was even having major performance issues with ZFS over USB (USB 3.0 mind you). Transfers would stall for a good 15 seconds, every few seconds, and the transfer rates would never go above about 10MB/s. Eventually, the OS would stop responding, and I'd either barely manage a normal reboot, or have to force the issue. The moment I stopped using ZFS on USB-attached storage, the problems went away, and it now works great.

With or without Core Storage, HFS Plus is required to boot OS X.

Right, but I suppose I was imagining a better world where Core Storage was more than just an LVM, but rather a full-fledged filesystem. It has some great foundations. HFS+ itself is already benefitting from some of them when running on top of it. I foresee that Apple may someday strip down HFS+ a bit, and move some of its features over to Core Storage, where they can be far better implemented. Compression at the block-level for example. Time will tell I suppose.

 

[2984839]

With a well-designed operating system it's really not difficult to get what's required from ZFS.



Where the operating system makes appropriate use of the file system: ZFS is good for casual, non-tech users.



I'm never outraged by ZFS. Maybe because I had a good introduction to it.

Without a good introduction, any file system might seem unfriendly. I can throw all readers here in at the deep end with TN1150; someone can describe that as outrageously complex.

I'm referring to the developer side of things. ZFS is a very large, complex codebase with a lot of features that is much larger than alternatives. Porting it is not straightforward and it would be easy to introduce bugs.

 

[grahamperrin]

https://github.com/openzfsonosx/zfs/issues/293#issuecomment-137557822 outlines the enhancement request that I made whilst I was with the AppleSeed programme. Not rushed; I tested ZEVO for a year or so (fitness for purpose) before making the more generic request for OS X to include support for ZFS.

The cases of OS X not responding, at least for me, were 100% when I was using it with a USB-attached device of some sort, …

If https://openzfsonosx.org/forum/viewtopic.php?f=26&t=2373 was removed you'll find a capture at http://web.archive.org/web/20151107061126/https://openzfsonosx.org/forum/viewtopic.php?f=26&t=2373 – SSHD, no USB, Core Storage encrypted ZFS home directory.

ZFS is a very large, complex codebase with a lot of features that is much larger than alternatives. Porting it is not straightforward …

Consistency across platforms was amongst the questions raised in the keynote at last year's OpenZFS Developer Summit. Also from the keynote:

Currently open in the ZFS on Linux area: sixty-two major bugs, two blockers. Some of those bugs will not affect users of OpenZFS on OS X; some of the OpenZFS on OS X ZFS issues will not affect users of ZFS on Linux; and so on.

On the OS X side, significant enhancements that are not yet closed include https://github.com/openzfsonosx/zfs/issues/116 related to permanent version storage and a variety of other issues.

 

[Markoth]

If https://openzfsonosx.org/forum/viewtopic.php?f=26&t=2373 was removed you'll find a capture at http://web.archive.org/web/20151107061126/https://openzfsonosx.org/forum/viewtopic.php?f=26&t=2373 – SSHD, no USB, Core Storage encrypted ZFS home directory.

Although, he doesn't say whether or not he was using an external (or internal via the SD card slot) ZFS-formatted disk at the time. For me, it would render the entire system unresponsive, regardless of which disk I was trying to access. If I had ANY ZFS-formatted USB drive, the system would eventually grind to a halt. I'm guessing he may have had something else formatted as ZFS. It isn't an obvious bug, but now that I've identified it, it's easily reproducible.

 

[grahamperrin]

He = me and I'm almost certain that none of those incidents involved a peripheral drive.

The later write-up https://fuzzy.wordpress.com/2015/11/07/os-x-10-9-5-stops-responding-with-openzfs-on-os-x-1-4-5/ did involve a peripheral drive.

 

[antonis]

Regardless the OS X itself, surely lots of things will break by moving to a case-sensitive filesystem. I recall a lot of 3rd party s/w stating in their requirements that they will only work on an HFS+ case-insensitive filesystem (remember, OS X partitioning also offers a case-sensitive option for HFS+). I'm sure even the most part of the rest of 3d party s/w that do not state a specific preference regarding the filesystem, they take as a given that they will work on HFS+ and will have problems under any other fs without modifications.

Having said that, however, I also believe Apple should make the move towards a more modern/advanced fs.

 

[thermodynamic]

Is it cheaper for Apple to make a new standard or corral people to existing ones? Is there enough of a market? Despite a "pro" line of hardware (at least for the little round desktops that don't dissipate heat nearly as well as the previous design form factor), Apple has to service a need or make a need to service. Looks like limiting on-board storage and pressing people to go to their server farms (I mean, iCloud is the fourth or fifth iteration of their hosted storage service, since they couldn't figure out a way to sing and dance to get the previous iterations to be popular and local storage is cheap - just get around standard I/O ports and lock the designs so customers can't do affordable post-purchase upgrades.

If Apple is not going to make ZFS, then OpenZFS or another solution is necessary.

 

[thermodynamic]

Well, personally, I've been using OpenZFS on OS X for awhile now, and it's working great! Just don't use it with a USB-attached drive (including the SD Card slot). Seems there are still some unresolved issues there. All-in-all, the space I'm saving from ZFS's compression feature has been well worth it! I'm using a combination of GZIP-9 (for when performance isn't an issue) and LZ4 (which tends to actually result in a performance boost). I've gained somewhere around the area of an extra 30GB on this 128GB internal SSD, just from ZFS. I don't really care if Apple implements something better at this point. ZFS is working just fine. The only downside is you can't boot from ZFS, so the boot partition still has to be either HFS+ or Core Storage.

Which brings up another point. Core Storage IS the future of HFS+. It has all the signs of someday becoming what ZFS is today. Snapshots, RAID and compression could easily become features of Core Storage in the future. Whether it will happen depends entirely on Apple, but the foundation is definitely there.



Exactly. The goal is to have the user NOT notice the filesystem. When a computer malfunctions less, the user ends up being happier, but they couldn't tell you why. All they know is they're NOT having issues. That's the goal of a filesystem: To do its job well. If a filesystem can't do that (HFS+ for example), then it gets noticed.

And if it's noticed, it's in the form of corrupt or missing files... much fun is had by all.

But people say the same thing about God(s). God(s) didn't invent all these human inventions, so it's up to be corporate responsible since they want corporate personhood. Unless Persephone and her buddy, Centeocihuatl the Corn God, created all this tech when they got bored once the harvest was over... Which goes back to the core question, why Apple dumped ZFS, won't make compression in HFS easily accessible (compression feature was added in 10.6), etc...)