why do you always use the wayback page, when the information is on apple.com. I find the wayback to be slower and at times, not the entire page.
Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Introducing Apple File System (APFS)
- Thread starter robE89
- Start date
- Sort by reaction score
Not always. I sometimes use it when there's certainty, or likelihood, of changes to the content and/or URL.
Please, let's keep this focused on discussion of AFPS and related file systems; not on people's styles of writing.
I'm not commenting on your writing style but rather the source of the information. I find thta posting a direct link does not obfuscate the discussion. With a direct link to apple its becomes more clearer which actually aides the discussion.
[doublepost=1466081750][/doublepost]
I'm sure there's quite a bit of loose ends in the beta that need to be tied up.
Yes, I deal with Oracle all the time, thanks to their database, Fusion applications and PeopleSoft. Its not fun by any stretch of the imagination. I agree that Apple can roll out a file system that is attuned to today's needs, for single user computing, instead of using or altering a FS that was developed for servers.
Spotlight and SearchFS
SearchFS is one of three things that APFS will not support.
An Apple Developer retired document: searchfs(2) Mac OS X Developer Tools Manual Page
rdar://9991647: Sandbox entitlements: need support for FSCatalogSearch and searchfs (2011-08-20)
VNOP_SEARCHFS · Issue #100 · openzfsonosx/zfs (2013-11-28) – fairly non-trivial.
Last but not least, from FS Search Xsan 2.2 | Xsanity (2009-09-22):
SearchFS is one of three things that APFS will not support.
An Apple Developer retired document: searchfs(2) Mac OS X Developer Tools Manual Page
rdar://9991647: Sandbox entitlements: need support for FSCatalogSearch and searchfs (2011-08-20)
VNOP_SEARCHFS · Issue #100 · openzfsonosx/zfs (2013-11-28) – fairly non-trivial.
Last but not least, from FS Search Xsan 2.2 | Xsanity (2009-09-22):
Now – without focusing on Xsan – think about encrypted metadata and user-friendly indexing of data that is (or will become) non-local through Apple's optimised storage.
I'm kind of warming to the file-system; I mean, I'd have preferred it if Apple had forked BTRFS and added encryption and other improvements to that, since BTRFS has some major performance benefits over ZFS while also being a fully copy-on-write file system (which is a big benefit for snapshots, I'm not clear on how APFS is handling these when only filesystem metadata is mentioned as being copy-on-write).
Other than Snapshots, which I assume are destined for a better version of Time Machine, I'd say Space Sharing is one of the most interesting improvements. It means that each user account can be on its own file-system while sharing the space of a single disk (maybe we'll get some kind of quota support too?), this means accounts can be encrypted individually. Not sure whether Time Machine will backup the encrypted data with some awareness of which blocks were changed at that level, or if maybe the user's key will remain available after they logout so the system can run a quick backup of their account before discarding the key till they log back in (since if they're logged out it's unlikely anything in their folder will be changing
)
This is also good because it means that an erase and install needn't wipe user data anymore if the OS itself goes into its own file-system as well.
Hell, it could greatly simplify containers too if there's some way to put apps into individual file-systems yet let them see files from another (read-only) one. Could be good for security too if the entire OS file-system is made read-only with a separate location for cache folders and such.
Just thinking out loud, but the ability to add file-systems into a shared space has a lot of potential, even if we have to set some of it up ourselves.
Other than Snapshots, which I assume are destined for a better version of Time Machine, I'd say Space Sharing is one of the most interesting improvements. It means that each user account can be on its own file-system while sharing the space of a single disk (maybe we'll get some kind of quota support too?), this means accounts can be encrypted individually. Not sure whether Time Machine will backup the encrypted data with some awareness of which blocks were changed at that level, or if maybe the user's key will remain available after they logout so the system can run a quick backup of their account before discarding the key till they log back in (since if they're logged out it's unlikely anything in their folder will be changing
)This is also good because it means that an erase and install needn't wipe user data anymore if the OS itself goes into its own file-system as well.
Hell, it could greatly simplify containers too if there's some way to put apps into individual file-systems yet let them see files from another (read-only) one. Could be good for security too if the entire OS file-system is made read-only with a separate location for cache folders and such.
Just thinking out loud, but the ability to add file-systems into a shared space has a lot of potential, even if we have to set some of it up ourselves.
As far as I recall, CoW in APFS is not limited to metadata. If it's not clear in the 160-page PDF from WWDC Session 701 then it should be clear from listening to Giampaolo and Tamura.
It may help to get your head around:
- per-extent encryption; and
- the singular approach to metadata when multi-key encryption is used; see around page 119 of the PDF.
When macOS gains the ability to boot from APFS (I should not expect that from any 2016 release, assume that it'll be a feature of 10.13) I expect that a variety of file systems, with diverse qualities, will be mounted to form a hierarchy of folders that will be familiar to users of the operating system.
Consider the proven PC-BSD approach –
Code:
$ freebsd-version ; uname -a ; beadm list
11.0-CURRENTJUNE2016
FreeBSD hpelitebook850g2-pcbsd.university.brighton.ac.uk 11.0-CURRENTJUNE2016 FreeBSD 11.0-CURRENTJUNE2016 #3 ee1d9a2(linux-drm-4.6): Wed Jun 8 16:52:23 UTC 2016 root@devastator:/usr/obj/usr/src/sys/GENERIC amd64
BE Active Mountpoint Space Created Nickname
11.0-CURRENTMAY2016-up-20160531_082033 - - 47.8M 2016-05-31 08:20 20160531_082010
11.0-CURRENTJUNE2016-up-20160609_161213 - - 4.2G 2016-06-09 16:10 11.0-CURRENTJUNE2016-up-20160609_161213
11.0-CURRENTJUNE2016-up-20160609_165742 NR / 20.0G 2016-06-09 16:56 11.0-CURRENTJUNE2016-up-20160609_165742
11.0-CURRENTJUNE2016-up-20160610_163846 - - 972.0M 2016-06-10 16:37 11.0-CURRENTJUNE2016-up-20160610_163846
11.0-CURRENTJUNE2016-up-20160616_004107 - - 834.0M 2016-06-16 00:38 11.0-CURRENTJUNE2016-up-20160616_004107
11.0-CURRENTJUNE2016-up-20160616_091805 - - 5.4G 2016-06-16 09:15 11.0-CURRENTJUNE2016-up-20160616_091805
$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
persona_gjp4 12.5G 15.9G 10.7G /usr/home/gjp4
tank 52.9G 838G 96K none
tank/ROOT 31.1G 838G 96K none
tank/ROOT/11.0-CURRENTJUNE2016-up-20160609_161213 4.12G 838G 12.1G /
tank/ROOT/11.0-CURRENTJUNE2016-up-20160609_165742 20.0G 838G 14.5G /
tank/ROOT/11.0-CURRENTJUNE2016-up-20160610_163846 824M 838G 13.6G /
tank/ROOT/11.0-CURRENTJUNE2016-up-20160616_004107 824M 838G 14.4G /
tank/ROOT/11.0-CURRENTJUNE2016-up-20160616_091805 5.30G 838G 13.6G /
tank/ROOT/11.0-CURRENTMAY2016-up-20160531_082033 8.95M 838G 13.8G /
tank/tmp 388K 838G 388K /tmp
tank/usr 13.4G 838G 96K none
tank/usr/home 2.57G 838G 96K /usr/home
tank/usr/home/bbsadmin-l 2.56G 838G 2.56G /usr/home/bbsadmin-l
tank/usr/home/gjp4 4.16M 838G 4.16M /usr/home/gjp4
tank/usr/jails 96K 838G 96K /usr/jails
tank/usr/local 317M 838G 96K none
tank/usr/local/share 317M 838G 96K none
tank/usr/local/share/doc 317M 838G 317M /usr/local/share/doc
tank/usr/obj 96K 838G 96K /usr/obj
tank/usr/ports 10.5G 838G 10.5G /usr/ports
tank/usr/src 96K 838G 96K /usr/src
tank/var 8.37G 838G 96K none
tank/var/audit 96K 838G 96K /var/audit
tank/var/log 469M 838G 469M /var/log
tank/var/mail 8.46M 838G 8.46M /var/mail
tank/var/tmp 7.90G 838G 7.90G /var/tmp
$That's six boot environments (BEs), four of which are significantly bugged so I currently prefer the second one that was created on 9th June.
The output from 'zfs list' puts those BEs in context.
Thanks! Given all the talk about crash protection, transactions, safe-save for directories, CoW and so on … I spent the last three days assuming that APFS would be reslient by design. (Not as resilient as a well-configured ZFS pool, but more resilient than Microsoft's default for ReFS … if that makes sense.)
So this came as a surprise:
fsck_apfs (APFS File System Check/Repair)
https://forums.developer.apple.com/thread/49032#144584 a mention of fsck_apfs finding errors with a volume.
From a more recent example:
Code:
darkstar:~ root# fsck_apfs /dev/disk4s2s1
** Checking volume.
** Checking the container superblock.
** Checking the space manager.
** Checking the object map.
** Checking the APFS volume superblock.
** Checking the object map.
** Checking the fsroot tree.
** Checking the extent ref tree.
** Checking the snapshot metadata tree.
** Checking the snapshots.
fsck_apfs: btree: invalid btn_btree.bt_longest_val (0), given btn_btree.bt_fixed.bt_val_size
Space Verification failed.
** The volume /dev/disk4s2s1 could not be verified completely.I wonder whether things such as checks of integrity and avoidance of errors will fall under the umbrella of extensibility …
Last edited:
APFS in Detail by Adam Leventhal:
I'll edit this post to include quotes, thoughts and observations.
Open source
Open source: doubts and P-words
Leventhal:
Don Brady (one of the people who were invited to comment on Leventhal's draft):
I imagine eyebrow-raising Private comments, in closed source, around Peculiarities of some of the things that expect HFS+ … and at least one such thing is an Apple technology. At this time I'll not name that thing; someone else might.
All things considered: whilst I would like Apple to prove us wrong, I don't expect the company to make open its source code for APFS. Time will tell.
(My imagination is sometimes too wild. When I look at a part of the code for OpenZFS that might have included a vaguely derogatory comment about a peculiarity of that Apple technology, neither that code nor its commit raises an eyebrow. There's also no reference to my related bug report, which makes the commit both professional and concise.
As lundman does open, professional and concise so I hope that Apple will combine those three things …)
- Overview
- Encryption, Snapshots, and Backup
- Space Efficiency and Clones
- Performance
- Data Integrity
- Conclusions
Open source
Open source: doubts and P-words
Leventhal:
Don Brady (one of the people who were invited to comment on Leventhal's draft):
I imagine eyebrow-raising Private comments, in closed source, around Peculiarities of some of the things that expect HFS+ … and at least one such thing is an Apple technology. At this time I'll not name that thing; someone else might.
All things considered: whilst I would like Apple to prove us wrong, I don't expect the company to make open its source code for APFS. Time will tell.
(My imagination is sometimes too wild. When I look at a part of the code for OpenZFS that might have included a vaguely derogatory comment about a peculiarity of that Apple technology, neither that code nor its commit raises an eyebrow. There's also no reference to my related bug report, which makes the commit both professional and concise.
As lundman does open, professional and concise
so I hope that Apple will combine those three things …)
Last edited:
U'r always gonna see this hand-in-hand type File-system
SSD's come along to replace hard drives - While an advantage, the next OS which has full support for SSD as a standard will be optimized for flash storage.... and faster (presume) noticeable, than one OS (like Yosemite) that is not.
Weather they'll be tests proving that, I think they will be tests done..
You mean like dragging files in and out of images, .pkg files etc and/or create files/folders inside of already created images?
This, i assume could mean the images could be dynamically resized on the fly based on the content added or deleted within them without trashing everything and creating a new image from scratch. Which could be very time consuming if u'r just make and 10+ Gig image size and forgot to add one extra folder.
SSD's come along to replace hard drives - While an advantage, the next OS which has full support for SSD as a standard will be optimized for flash storage.... and faster (presume) noticeable, than one OS (like Yosemite) that is not.
Weather they'll be tests proving that, I think they will be tests done..
You mean like dragging files in and out of images, .pkg files etc and/or create files/folders inside of already created images?
This, i assume could mean the images could be dynamically resized on the fly based on the content added or deleted within them without trashing everything and creating a new image from scratch. Which could be very time consuming if u'r just make and 10+ Gig image size and forgot to add one extra folder.
Last edited:
That's not a necessity. Recall that Apple ran ZFS on iPhone in 2009 when (according to Wikipedia) the 3GS had 256 MB DRAM.
+1
Imagine a mass market (of Mac users) gaining the ability to discover – with ease – corruption of data on disks that were previously apparently OK. The ire of some of those users.
It's possible for a drive to appear error-free with a check such as this –
– but an error-free check of all blocks is not a guarantee that data is free from errors.
That's not a criticism of Drive Genius. It's to emphasise the value of checksummed data.
That's not a necessity. Recall that Apple ran ZFS on iPhone in 2009 when (according to Wikipedia) the 3GS had 256 MB DRAM.
+1
Imagine a mass market (of Mac users) gaining the ability to discover – with ease – corruption of data on disks that were previously apparently OK. The ire of some of those users.
It's possible for a drive to appear error-free with a check such as this –
– but an error-free check of all blocks is not a guarantee that data is free from errors.
That's not a criticism of Drive Genius. It's to emphasise the value of checksummed data.
I can't imagine it.
I'm not saying you're wrong, but I had no idea ZFS ran on the old iPhones (back in 2009 I was deep in Solaris-user land: no Mac, no iPhone)! I do remember ZFS being on Mac OS Leopard though; I even played with it a bit on my wife's MacBook Pro.
Question. How would Apple (or how does ZFS) implement data integrity checks on machines with only a single physical drive? And even if the integrity check comes up bad how would you correct the data with only one drive?
IMO as long as these features require some kind of RAID like setup, I can't see it being a high priority for Apple. The only thing I could think of would be to do checks via some future version of time machine although I have no idea how it would work.
IMO as long as these features require some kind of RAID like setup, I can't see it being a high priority for Apple. The only thing I could think of would be to do checks via some future version of time machine although I have no idea how it would work.
@Tech198 please, what can you not imagine? (There were multiple points in what you quoted.)
Adam Leventhal's blog » ZFS: Apple’s New Filesystem That Wasn’t – mentioned in his blog post, and elsewhere.
Metadata: redundancy is the norm, and may allow repairs to be automated during a routine scrub. I have a very recent example (for now, I refrain from sharing the details).
Data: the number of copies can be 1, 2 or 3. Plus snapshots, and a common sense backup routine that stores data away from the single drive. Where that common sense applies: 1 should suffice, no copies. https://duckduckgo.com/?q="ZFS+set+copies=1" leads to some pages where copies are described as ditto blocks.
If there's no recent backup, then local snapshots may include a recent, scrubbed, error-free version of the data.
It's a glaring omission – "ire to data integrity" and so on.
Last edited:
During the session they said apfs will be released open source when it's finished in 2017.
Again, during the session they mentioned why they didn't use ZFS - but it seems very similar. I think - based on the coding sessions - that the programming patterns they're encouraging developers adopt would be optimized by the apfs. There's a also a few differences with ZFS, but they're not trying to create a general fs that runs on 3rd party hardware. They're designing this for their own hardware and software - not optimized for generic use cases - so it's different, like everything else...
First, thanks for the detailed and informative reply. It's great to know that integrity could be achieved without relying on raid like technologies.
I do wish you hadn't missed quoted me there at the end though.
If you reread the sentence in full, I think it's obvious that I meant that IF data integrity relies on RAID like technologies, I can't see it being a high priority for Apple (because most of their machines/devices don't ship in a RAID capable configuration, and most consumers don't want that kind of complexity).
If you're correct and it does not require this kind of setup then I would agree that it does seem like a rather disappointing omission AT THIS TIME. As there is still a year before release and we haven't seen the new backup solution I think there's still plenty of potential for it to come up
RAID and FS-level integrity checking have nothing to do with each other. RAID is usually an additional, lower level (although some FS implement RAID as part of their features — but we both can agree that its too much unnecessary complexity). I am running several RAID storage units with our Mac Server and i'd love to have some additional data protection in a form of FS-level autocorrecting codes. The more protection layers there is the better.
My apologies to all readers, I did not intend to misleadlead or mis-quote. (With the ellipses '…' and the link, I imagined that people would read the linked original). @Malus120 I have edited the quote to include both ends of your sentence.
I recall talk of things such as documentation becoming open but – unless I missed something – no clear indication, from Apple, that the source code will be open. From Leventhal's conclusion: "… I don’t expect APFS to be open source at this time or any other, but prove me wrong, …"; and Giampaolo thanked Leventhal for even-handedness and correctness.
Sierra
El Capitan
From a topic about future updates to El Capitan, with the author's emphasis on APFS and El Capitan:
That's the current situation.
Some milestones
Possible chronological order:
I expect 4, 5 and 6 to be broadly coincidental – around the same time of year, not necessarily the same date.
For the operating systems that are commonly associated with Apple
I can't imagine APFS as a default for any release of iOS, tvOS or watchOS before it becomes the default for a release of macOS. Whilst the file system might become release-quality for any/all of the first three before macOS, strategically it will make sense for Apple to not formally release APFS until it's perceived to be of suitable quality for all four operating systems. (Some Mac users are pleasingly difficult to please … to be treated as second-, third- or fourth-rate after waiting so long for a good storage system would cause a mild outcry.)
Those four aside
http://apple.stackexchange.com/a/106026/8546 shows that Time Capsule runs NetBSD.
Possible chronological order:
- APFS developer preview in pre-release 10.12
- APFS developer preview in the first release of 10.12
- APFS developer preview or beta in pre-release 10.13, to include boot from APFS
- APFS in the first release of 10.13
- APFS to ship (first release) in 2017
- APFS to be the default file system for all Apple products, although I don't yet know whether, or how, Time Capsule will transition to APFS
- 10.11.x to gain support for APFS data volumes.
I expect 4, 5 and 6 to be broadly coincidental – around the same time of year, not necessarily the same date.
For the operating systems that are commonly associated with Apple
I can't imagine APFS as a default for any release of iOS, tvOS or watchOS before it becomes the default for a release of macOS. Whilst the file system might become release-quality for any/all of the first three before macOS, strategically it will make sense for Apple to not formally release APFS until it's perceived to be of suitable quality for all four operating systems. (Some Mac users are pleasingly difficult to please … to be treated as second-, third- or fourth-rate after waiting so long for a good storage system would cause a mild outcry.)
Those four aside
http://apple.stackexchange.com/a/106026/8546 shows that Time Capsule runs NetBSD.
The time capsule doesn't need to directly support APFS, given that the computer being backed up creates a disk image on the Time Capsule's disk. The internal format of the disk image is invisible to the Time Capsule and so as long as macOS continues to support backup via AFP, there should be no change needed for a Time Capsule. Alternatively, it's not difficult to imagine that Apple simply opts to make all existing Time Capsules obsolete by the time 10.13 ships. Given that a transition toward completely removing AFP is underway, this would not be particularly surprising.
Given previous patterns, Apple will transition 10.11.x to security updates only next month when 10.12 is released. HFS+ support is very likely to remain in macOS for many years; what possible reason would lead you to believe that 10.11.x would receive APFS support? (With Apple providing written documentation as quoted above that APFS support is not in 10.11.x, the verbal statement made at WWDC regarding the lack of APFS support in 10.10.x and older must be considered an inadvertent misstatement.) The oldest computers which can run 10.12 is currently 7 years old, and will be 8 years old when 10.13 is released. Apple will never back port APFS to computers which are unable to run 10.12.
When Apple introduced HFS+ with Mac OS 8.1, there was no provision for Macs which couldn't run 8.1 to access these volumes and it's only recently that support for original HFS volumes was removed from OS X. Precedents are established for Apple's handling of file system changes.
Last edited:
Why would they have to if APFS goes open source? Surely it will become available anyway quite quickly from people with the interest in porting it.
That's probably the greatest unknown, and I'm with Leventhal – don’t expect APFS to be open source at this time or any other, but prove me wrong, Apple.
Imagine a future Time Machine server device from Apple running an operating system other than NetBSD; running an OS with native support for APFS.
If that will happen then (as with OS X Server) there will be no need for the Mac OS X client to use hdiutil(1).
Nothing written or spoken by Apple (or shouted in red text) goes against the expectation that 10.11.x will gain limited support for APFS.
Anyone:
- imagine being a user of one of the thousands of Macs that will not run 10.12
- also, imagine running 10.10.x on that Mac
- finally, imagine some time in 2017, receiving a disk that was formatted by 10.13.
a) dislike the incompatibility (and maybe have notions of planned obsolescence by Apple); or
b) accept Apple's free upgrade to 10.11 …
What is relevant to the backup process is the communication protocol, not the file system. Having native support for APFS does not mean that the system supports TM backups.