Introducing Apple File System (APFS)

[bradl]

I hate Apple creating its own standards/technologies that relate to computing these days...

Then we should all expect you to dump all Apple gear and go back to using an 8088 or older processor;

the 64bit architecture was designed by AMD - its own standards/technologies; which was adopted by Intel.

the x86 architecture was designed by Intel - its own standards/technologies; something that AMD, Cyrix, and others adopted.

The GUI in OS X itself was designed by X11, which goes back for debate between ATT/Bell Labs and SysV; something that Apple, Microsoft, Sun, NeXT, DEC, HP, IBM, and many others have adopted.

At that point, that eliminates your use of any major tech, because all of it was adopted by someone who created their own standards and technologies that relate to computing.

BL.

 

[newfoundglory]

WARNING:  You are using a pre-release version of the Apple File System called
          APFS which is meant for evaluation and development purposes only.
          Files stored on APFS volumes may not be accessible in future releases
          of OS X.  You should back up all of your data before using APFS and
          regularly back up data while using APFS, including before upgrading
          to future releases of OS X.

You can pass the "-IHaveBeenWarnedThatAPFSIsPreReleaseAndThatIMayLoseData"
option between the "APFS" verb and the APFS sub-verb to bypass this message.
Proceed? (y/N)

[doublepost=1465930873][/doublepost]Seems like they forgot to change OS X to macOS in the warning

 

[grahamperrin]

@newfoundglory thanks, I sent feedback.

 

[Mac 128]

A new filesystem is loooong overdue, bit of a shame Apple chose to reinvent the wheel rather than adopt ZFS. This was the perfect opportunity to switch to ZFS, and it looks like they're gonna blow it. Arrogance and NIH syndrome. Kinda sad, really...
[doublepost=1465917294][/doublepost]

Agreed; it's a shame that Apple didn't adopt ZFS. I don't trust my data to ANY other file system.

I miss MFS. ;-)

 

[RichTeer]

Maybe the FBI/NSF has a backdoor into ZFS's encryption scheme and Apple wants to ensure this doesn't propagate into macOS.

I will admit that's a (slim) possibility for Oracle's version of ZFS, it certainly isn't for OpenZFS' encryption support: the whole thing is open source so anyone can see for themselves.

 

[grahamperrin]

@Mac 128 I assume that APFS will have appropriate support for POSIX filesystem semantics, and – joking aside – Apple's optimised storage may make it easier for some customers to take a flat file system view of things.

For any youngsters who have never heard of MFS: https://en.wikipedia.org/wiki/Macintosh_File_System (when I began using Macs, MFS was outdated but still readable where System 7 was installed; the installers on one-sided MFS floppy disks were a curiosity).

 

[Zirel]

I hate Apple creating its own standards/technologies that relate to computing these days... I honestly believe Metal is the nail in the coffin for AAA gaming on the Mac.

Presume you love what's happening on Windows and Linux, then?

Microsoft file systems are closed source, and everything on OS X and Linux is reverse engineering. Notwithstanding, they gave up on their COW filesystem.

On Linux, yes, it's "open source", but you can only use it in GPL operating systems, which is brilliant, because Apple cannot use it then. But they turn a blind eye to Google and all the Android OEMs, because it's Google, Samsung and IBM who are doing so much of the heavy lifting in Linux. Also, btrfs is always in testing, and countless others have been basically abandoned.

ZFS has nice features, but it's a RAM hog, and most feature are irrelevant in SSD's, and for single users. Also, dealing with Oracle can be troublesome, ask Google.

Also, APFS is going to be open source, so there's that. Which still is irrelevant, as sharing files through pen drives will completely die sooner than later, as more and more people are using cloud and network drives.

 

[RichTeer]

ZFS has nice features, but it's a RAM hog, and most feature are irrelevant in SSD's, and for single users. Also, dealing with Oracle can be troublesome, ask Google.

Yes, ZFS likes lots of memory, but I don't think features like end-to-end checksums (for assuring the correctness of one's data) are irrelevant for SSDs or single users. I don't know about you, but I highly value the integrity of my data! And with OpenZFS, Apple needn't deal with Oracle...

 

[grahamperrin]

Tuesday File Systems Lab Notes | Apple Developer Forums

Postscripts

In response to https://forums.macrumors.com/posts/23029711 under Revert the core storage!

Question:
How does CoreStorage interact with APFS during the APFS conversion process?

Answer:
The plan is to have the APFS conversion process read the Core Storage plus HFS+ metadata and convert them to an APFS container. For encrypted Core Storage volumes, you will likely be prompted for an account password or recovery key before the conversion process begins.

– so, looking ahead:

• where Core Storage is a default, it may be prudent to accept that default.

One very brief test a few hours ago. Early indications are that:

• I read to much into the answer without paying attention to the question (sorry!)
  
• conversion can occur without Core Storage.

 

[jameslmoser]

Good question. It's irrelevant to the file system. Equally off-topic:

What about using a hard drive (or drives) over TB3?? Filesystems might be relevant there! ;-)

 

[grahamperrin]

@jameslmoser haha although if we respect the original context, an elephant in the room question must be asked: will AAA APFS allow conversion from Core Storage to nVidia?

 

[maflynn]

Finally a new(modern) filesystem!

I need to read the details, to what it includes. I actually missed it in the keynote, I thought they were talking more about iCloud stuff.

Anyways, I'm hoping it does offer some nice improvments

I hate Apple creating its own standards/technologies that relate to computing these days... I honestly believe Metal is the nail in the coffin for AAA gaming on the Mac.

Apple has a long history of going their own route, Just consider ADB, or Nubus, or firewire or 3.5 floppies, or even more recenly lightening connector.

In rare situations what apple picked, became standard (I'm looking at you 3.5" floppy), but most they took (or developed) non-standard technology, incorporated that into their designs and used it for years.

 

[maflynn]

I'm listening to MacBreak Weekly, and they're stating that APFS is not going to included in MacOS Sierra, but a later version of the OS.

 

[grahamperrin]

APFS

The video of Session 701 – Introducing Apple File System – is now available to play in Safari (not compatible with browsers such as Firefox or Google Chrome). Around thirty-six minutes.

… will be available as a developer preview technology in macOS Sierra once it finally ships this fall. …

The data structures in HFS Plus are … relatively single-threaded. So our B-trees rely on a big lock in order to access or mutate them. …

Why a new file system?

Designed (and tuned) for Apple products and ecosystem … we typically use a single storage device on all of our products. …

… often a tradeoff between latency and throughput, and we have chosen to lean on the side of latency. …

… will support and replace HFS Plus functionality … will not support … exchange data … SearchFS … directory hard links …

(… spontaneous applause after the explanation of atomic safe-save (renamex_np) …)​

… multi-key encryption. Here, all sensitive metadata is encrypted with a single key that's distinct from the per-file keys …

… per-extent encryption so each region of a file can be encrypted with its own key. This is unique, and no other file system out there supports anything like this. …

(… spontaneous applause for provision of an in-place upgrade path …)​

Encryption of all sensitive metadata with a single key might fit with my Spotlight-related thoughts in the Location independent files topic.

ZFS

ZFS … RAM hog, and most feature are irrelevant in SSD's, and for single users …

Such preconceptions are commonplace.

Last year I was frequently the single user of an all-ZFS system on a 2007 notebook with 2 GB memory.

Relevance to solid state? For starters, there's Intel's co-sponsorship of the 2016 OpenZFS Developer Summit.

 

[Zirel]

Relevance to solid state? For starters, there's Intel's co-sponsorship of the 2016 OpenZFS Developer Summit.

Great argument! /s

 

[grahamperrin]

… MacBreak Weekly… stating that APFS is not going to included in MacOS Sierra …

I guess it's open to interpretation.

If I heard correctly (partial transcript above), at WWDC Eric Tamura emphasised the word "in" when describing the developer preview technology in the forthcoming release of Sierra.

 

[Thunderbird]

Apple made an announcement that they are working on a new file system to replace HFS+ and there have been some speculations and write ups about it:

http://appleinsider.com/articles/16...ll-scale-from-apple-watch-to-macs-replace-hfs

http://9to5mac.com/2016/06/13/apple-file-system-apfs/

http://arstechnica.com/apple/2016/0...ocumentation-for-apfs-apples-new-file-system/

My questions are:

1. Is this for real, or is it going to be another ZFS fiasco?

2. Will this affect Boot Camp? Will they need to write new drivers? Or will they just stop supporting Boot Camp?

3. How will this affect Time Machine?

4. Would we reformat our existing HFS+ drives via a download or perhaps an OS update?

 

[LauraJean]

Exactly how is a new file system implemented on a mac? Will every file on my hard drive be written somewhere to the new format and then the old format file be deleted? What kind of space is needed for such an operation? How much time per GB? And what about data stored on other media, like jump drives?

Being retired IT, I equate "file system" with "platform", and a "platform" change is a big deal. We called it "data conversion" and it usually occurred in tandem with a hardware/operating system change.

 

[MikhailT]

I'm listening to MacBreak Weekly, and they're stating that APFS is not going to included in MacOS Sierra, but a later version of the OS.

No, it is included in Sierra DP1 already, you can use command line to create the containers and play with it. However it will not be turned on by default, the file system lab session confirmed this as well. They want Sierra to be the first one that future versions of APFS containers can be used on.

The session also confirmed that Apple will convert everyone to APFS sometime next year.

You can see it here: https://developer.apple.com/videos/play/wwdc2016/701/

Exactly how is a new file system implemented on a mac? Will every file on my hard drive be written somewhere to the new format and then the old format file be deleted? What kind of space is needed for such an operation? How much time per GB? And what about data stored on other media, like jump drives?

Being retired IT, I equate "file system" with "platform", and a "platform" change is a big deal. We called it "data conversion" and it usually occurred in tandem with a hardware/operating system change.

They explained how the upgrade will take over in the session here: https://developer.apple.com/videos/play/wwdc2016/701/ Skip to 33min.

 

[grahamperrin]

@LauraJean

• https://forums.macrumors.com/posts/23009805 on page 1 begins with the the link to APFS-related Session 701 at WWDC
  
• https://forums.macrumors.com/posts/23016166 above links to an attendee's notes, which mention the conversion process
  
• https://forums.macrumors.com/posts/23019603 above includes a few words from that session, please note the spontaneous applause for provision of an in-place upgrade path.

The applause is understandable; listen from 32:40 on the timeline.

@maflynn I heard correct, now I see it on screen at that point on the timeline.

… They want Sierra to be the first one that future versions of APFS containers can be used on. …

Please: does that mean that whilst El Capitan will gain support for APFS, Apple will not (or can not) guarantee that El Capitan will gain support for all future extensions to the file system?

I did listen to the description of extensibility – comparable to ZFS pool feature flags, I guess – but when I first listened to the session I skipped nearly all of Eric Tamura's speech in the second half. It will be interesting to hear, but right now (past 01:30 in the UK) I don't have time.

 

[LauraJean]

@MikhailT, @grahamperrin

I watched all of video701 on the coming Apple File System and the post references. Thank you so much, those two Apple dudes were impressive--an "in-place" update! Despite being retired for 6 years, I, to my surprise, enjoyed the presentation with 99% understanding. It made me a little nostalgic for my working days when I was much more than the low-level end-user I've become.

 

[xoAnna]

Apple made an announcement that they are working on a new file system to replace HFS+ and there have been some speculations and write ups about it:

http://appleinsider.com/articles/16...ll-scale-from-apple-watch-to-macs-replace-hfs

http://9to5mac.com/2016/06/13/apple-file-system-apfs/

http://arstechnica.com/apple/2016/0...ocumentation-for-apfs-apples-new-file-system/

My questions are:

1. Is this for real, or is it going to be another ZFS fiasco?

2. Will this affect Boot Camp? Will they need to write new drivers? Or will they just stop supporting Boot Camp?

3. How will this affect Time Machine?

4. Would we reformat our existing HFS+ drives via a download or perhaps an OS update?

Ditto to all your questions above.

 

[grahamperrin]

1. Is this for real …?

Yes.

2. Will this affect Boot Camp?

I doubt it.

Expect Windows to boot, as it already does, from NTFS on part of a physical device without an APFS container.

3. How will this affect Time Machine?

Apple's current implementation of Time Machine requires directory hard links. See https://forums.macrumors.com/posts/23019603 above, expand the view to reveal all of what was quoted; Eric Tamura spoke of three things that APFS will not support.

Also please be aware of the previously linked materials, in particular the current limitations.

4. Would we reformat our existing HFS+ drives via a download or perhaps an OS update?

Neither. In https://forums.macrumors.com/posts/23019603 the quoted text drew attention to an upgrade path.

 

[maflynn]

1. Is this for real, or is it going to be another ZFS fiasco?

They announced it WWDC, so I'd say its for real.

2. Will this affect Boot Camp? Will they need to write new drivers? Or will they just stop supporting Boot Camp?

I think the reformatting process will be such that the windows partition will be left intact.

3. How will this affect Time Machine?

The backing up/versioning will be much more streamlined, given the features built into APFS

4. Would we reformat our existing HFS+ drives via a download or perhaps an OS update?

I'd say given the major impact a new FS will have, this will be done during an OS upgrade, not even update. I suspect there may be a wizard or app included to convert the existing drive, but overall, there's so much that needs to change that I think it be part of the next OS release next year
[doublepost=1466074729][/doublepost]

No, it is included in Sierra DP1 already,

but I don't think it's at a point where it will be ready for general consumption by this fall. I think its there for some to get used to it, but you cannot rush out a new FS. Its one thing to have a buggy Finder, its quite another to have a buggy file system.

 

[grahamperrin]

Introducing Apple File System

Session 701 slides in the Internet Archive Wayback Machine:

• https://web.archive.org/web/2016061...crv/701/701_introducing_apple_file_system.pdf (160 pages)

Apple File System converter

… given the major impact a new FS will have,

"… a multi-second to multi-minute operation potentially …" – Eric Tamura

this will be done during an OS upgrade, not even update …

For conversion of the macOS startup volume, I should expect an option in the installer for macOS 10.13.

Whether conversion will be the default in that context, I'll not guess at this time. There'll be crash protection (to safeguard data and metadata in the event of a crash) but it may be preferable to steer customers towards post-update conversion.

Paces of development and delivery

… you cannot rush out a new FS …

It will not be release quality when Sierra is released, but I doubt that its development was – or will be – rushed.

If I understand correctly:

• both multi-key encryption and Apple's unique per-extent encryption should complement cloud aspects of optimised storage, for which a related patent was filed in January 2012.

I assume that the Storage and File Systems teams began work on/around what's now known as APFS some time before 2012. Work on APFS began two years ago.