cMP Slow AFPS 4K Write performance

[handheldgames]

PCIE SSD's - gotta love the speed. But... Did you realize that using AFPS with a Fast PCIe SSD, is like pulling a boat anchor at the drag strip.

When it comes to measuring 4K performance on the cMP, AmorphiousDiskMark is the only MacOS App available and it happens to be FREE Looking at benchmarks from AmorphousDiskMark posted here on MacRumors and on the internet, I've noticed a HUGE discrepancy in 4K write performance. Roughly speaking, running with AFPS, write performance is around 50% slower than the same drive formatted with HFS.

For this test, I partitioned a 1TB 970 Pro to create a 2nd, HFS+ partition that was 125GB in size. After creating the partition, I ran 3 benchmarks on the HFS partition(top benchmarks) and one on the AFPS partition(bottom benchmark). The image below represents those 4 tests.

While I initially thought the issue could be with the Highpoint SSD7101a's Broadcom chipset, the slow 4k write issue points squarely at AFPS. Is anyone else seeing these results?

 

[AidenShaw]

PCIE SSD's - gotta love the speed. But... did your realize that using AFPS is like pulling a boat anchor at the drag strip.

When it comes to measuring 4K performance on the cMP, AmorphiousDiskMark is the only MacOS App available and it happens to be FREE Looking at benchmarks from AmorphousDiskMark posted here on MacRumors and on the internet, I've noticed a HUGE discrepancy in 4K write performance. Roughly speaking, running with AFPS, write performance is around 50% slower than the same drive formatted with HFS.

For this test, I partitioned a 1TB 970 Pro to create a 2nd, HFS+ partition that was 125GB in size. After creating the partition, I ran 3 benchmarks on the HFS partition(top benchmarks) and one on the AFPS partition(bottom benchmark). The image below represents those 4 tests.

While I initially thought the issue could be with the Highpoint SSD7101a's Broadcom chipset, the slow 4k write issue points squarely at AFPS. Is anyone else seeing these results?

View attachment 817211

It's rather hard to analyze based on fragments of screenshots. The read numbers seem entirely normal - everything within what would normally be expected for comparing short tests. The final write number seems a bit low, but without more tests and full disclosure of the parameters it is hard to tell.

Run 20 tests of each and post the means and standard deviations.

 

[h9826790]

PCIE SSD's - gotta love the speed. But... did your realize that using AFPS is like pulling a boat anchor at the drag strip.

When it comes to measuring 4K performance on the cMP, AmorphiousDiskMark is the only MacOS App available and it happens to be FREE Looking at benchmarks from AmorphousDiskMark posted here on MacRumors and on the internet, I've noticed a HUGE discrepancy in 4K write performance. Roughly speaking, running with AFPS, write performance is around 50% slower than the same drive formatted with HFS.

For this test, I partitioned a 1TB 970 Pro to create a 2nd, HFS+ partition that was 125GB in size. After creating the partition, I ran 3 benchmarks on the HFS partition(top benchmarks) and one on the AFPS partition(bottom benchmark). The image below represents those 4 tests.

While I initially thought the issue could be with the Highpoint SSD7101a's Broadcom chipset, the slow 4k write issue points squarely at AFPS. Is anyone else seeing these results?

View attachment 817211

I am quite sure AmorphousDiskMark is CPU single thread limiting on that 4K write test. You may check the CPU usage in Activity Monitor when running that test. But I am not sure if APFS is the reason to cause that CPU single thread limitation.

Anyway, you may use Xbench etc if you want something free and can measure 4K speed (as double check).

Lots of tests inside XBench are very outdated, but the storage speed test is actually OK.

 

[bsbeamer]

While I initially thought the issue could be with the Highpoint SSD7101a's Broadcom chipset, the slow 4k write issue points squarely at AFPS. Is anyone else seeing these results?

What OS version and build are you on?
Assume 140.0.0.0.0 firmware?

I ran into issues with APFS on authentic Mac Pro 5,1 during one of the High Sierra point updates. Believe that was 10.13.2 or 10.13.3 and documented it on this forum. This was ONLY with the APFS converted system drive, which was Samsung SATA SSD (likely EVO 850 or EVO 860) connected via Apricorn Velocity Solo X2. The "fix" was to clone system drive to an HFS+ formatted drive and replace. Immediately resolved all issues and write speeds went back to normal.

I still do not trust APFS with non-Apple issues SSDs, but there really are no other options if you move to Mojave.

 

[handheldgames]

What OS version and build are you on?
Assume 140.0.0.0.0 firmware?

I ran into issues with APFS on authentic Mac Pro 5,1 during one of the High Sierra point updates. Believe that was 10.13.2 or 10.13.3 and documented it on this forum. This was ONLY with the APFS converted system drive, which was Samsung SATA SSD (likely EVO 850 or EVO 860) connected via Apricorn Velocity Solo X2. The "fix" was to clone system drive to an HFS+ formatted drive and replace. Immediately resolved all issues and write speeds went back to normal.

I still do not trust APFS with non-Apple issues SSDs, but there really are no other options if you move to Mojave.

I'm running with EFI version 140.0.0.0.0 firmware as well. Both are running from a 970 Pro in a Highpoint SSD7101A.

Following up with some benchmark images that are more clear. In both instances, the AFPS drive on the left and the HFS Drive on the right. This should help to illustrate the performance slow down associated with AFPS...


Quickbench small file comparison

AmorphoeusDiskMark Comparison.

SATA II, as slow as it is, also appears to be effected. (apfs left / hfs right). And yes.. it looks like misspelled my partition name. Although it doesn't help the results.

 

[w1z]

I was able to replicate your results on multiple ssd storage systems ie. sata/pcie/nvme and believe it to be an issue with APFS as a file system unless newer macs that have the T2 chip don't suffer from it. I also tried to find ways of fine tuning sysctl apfs/file system values but only found apfs fusion/hfs related values to be tunable.

Can someone with a T2 enabled mac run these tests?

 

[bsbeamer]

Do not believe this has anything to do with T2 chips, but rather Apple issued SSDs vs non-Apple issues SSDs. My MacBookPro11,3 is now APFS (during High Sierra install) and does not run into any of these issues. It is well before the Tx chip days.

 

[w1z]

Do not believe this has anything to do with T2 chips, but rather Apple issued SSDs vs non-Apple issues SSDs. My MacBookPro11,3 is now APFS (during High Sierra install) and does not run into any of these issues. It is well before the Tx chip days.

Can you share your AmorphousDiskMark benchmarks?

The proprietary ssd controller on the new macs resides in the T2 coprocessor package and I suspect it is handling the NAND storage different than market controllers with APFS optimized for use with Apple's T2 ssd controller. I highly doubt Apple is using proprietary Samsung-made NAND chips. The NAND storage pcb can have proprietary connections but that's just it.

Only way to find out is to run the subject benchmarks on a T2 enabled mac.

 

[bsbeamer]

MacBookPro11,3
APPLE SSD SM1024F, UXM6JA1Q firmware, 1.0 TB, Apple (Samsung-based)

Note: Drive is VERY full, more than usual right now. Overall Read/Write on this MBP is usually higher than this. Has reported 750 MB/s+ in past.



If helpful, this is from DriveDx report:

### DRIVE 1 OF 1 ###
Last Checked : January 30, 2019 10:48:20 AM EST
Last Checked (ISO 8601 format) : 2019-01-30T10:48:20

Advanced SMART Status : OK
Overall Health Rating : GOOD 100%
SSD Lifetime Left Indicator : GOOD 100%
Issues found : 0

Serial Number :
WWN Id :
Volumes : Macintosh SSD
Device Path : /dev/disk0
Total Capacity : 1.0 TB (1,000,555,581,440 Bytes)
Model Family : Apple (Samsung-based) SSDs
Model : APPLE SSD SM1024F
Firmware Version : UXM6JA1Q
Drive Type : SSD

Power On Time : 11,537 hours (16 months 0 days 17 hours)
Power Cycles Count : 16,090
Current Power Cycle Time : 3.0 hours


=== DEVICE CAPABILITIES ===
S.M.A.R.T. support enabled : yes
DriveDx Active Diagnostic Config : Apple (Samsung-based) e/f-series SSDs config [ssd.apple.samsung.ef]
Sector Logical Size : 512
Sector Physical Size : 4096
Physical Interconnect : PCI
Logical Protocol : SATA
Removable : no
Ejectable : no
ATA Version : ATA8-ACS T13/1699-D revision 4c
SATA Version : SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
I/O Path : IOService:/AppleACPIPlatformExpert/PCI0@0/AppleACPIPCI/RP05@1C,4/IOPP/SSD0@0/AppleAHCI/PRT0@0/IOAHCIDevice@0/AppleAHCIDiskDriver/IOAHCIBlockStorageDevice
Attributes Data Structure Revision : 40
SMART Command Transport (SCT) flags : 0x0
SCT Status supported : no
SCT Feature Control supported : no
SCT Data Table supported : no
Error logging capabilities : 0x1
Self-tests supported : yes
Offline Data Collection capabilities : 0x5f
Offline Data Collection status : 0x0
Auto Offline Data Collection flags : 0x0
[Known device ]: yes
[Drive State Flags ]: 0x0

 

[w1z]

MacBookPro11,3
APPLE SSD SM1024F, UXM6JA1Q firmware, 1.0 TB, Apple (Samsung-based)

Note: Drive is VERY full, more than usual right now. Overall Read/Write on this MBP is usually higher than this. Has reported 750 MB/s+ in past.

View attachment 819012

If helpful, this is from DriveDx report:

### DRIVE 1 OF 1 ###
Last Checked : January 30, 2019 10:48:20 AM EST
Last Checked (ISO 8601 format) : 2019-01-30T10:48:20

Advanced SMART Status : OK
Overall Health Rating : GOOD 100%
SSD Lifetime Left Indicator : GOOD 100%
Issues found : 0

Serial Number :
WWN Id :
Volumes : Macintosh SSD
Device Path : /dev/disk0
Total Capacity : 1.0 TB (1,000,555,581,440 Bytes)
Model Family : Apple (Samsung-based) SSDs
Model : APPLE SSD SM1024F
Firmware Version : UXM6JA1Q
Drive Type : SSD

Power On Time : 11,537 hours (16 months 0 days 17 hours)
Power Cycles Count : 16,090
Current Power Cycle Time : 3.0 hours


=== DEVICE CAPABILITIES ===
S.M.A.R.T. support enabled : yes
DriveDx Active Diagnostic Config : Apple (Samsung-based) e/f-series SSDs config [ssd.apple.samsung.ef]
Sector Logical Size : 512
Sector Physical Size : 4096
Physical Interconnect : PCI
Logical Protocol : SATA
Removable : no
Ejectable : no
ATA Version : ATA8-ACS T13/1699-D revision 4c
SATA Version : SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
I/O Path : IOService:/AppleACPIPlatformExpert/PCI0@0/AppleACPIPCI/RP05@1C,4/IOPP/SSD0@0/AppleAHCI/PRT0@0/IOAHCIDevice@0/AppleAHCIDiskDriver/IOAHCIBlockStorageDevice
Attributes Data Structure Revision : 40
SMART Command Transport (SCT) flags : 0x0
SCT Status supported : no
SCT Feature Control supported : no
SCT Data Table supported : no
Error logging capabilities : 0x1
Self-tests supported : yes
Offline Data Collection capabilities : 0x5f
Offline Data Collection status : 0x0
Auto Offline Data Collection flags : 0x0
[Known device ]: yes
[Drive State Flags ]: 0x0

Your 4K read/write results are in line with the results posted in this thread which confirms that apfs 4K performance is impacted even on Apple SSD drives.

However, your MBP doesn't have the T2 chip and is most likely using a Samsung based SSD controller.

 

[bsbeamer]

Your 4K read/write results are in line with the results posted in this thread which confirms that apfs 4K performance is impacted even on Apple SSD drives.

However, your MBP doesn't have the T2 chip and is most likely using a Samsung based SSD controller.

As clearly noted, it is a Samsung-based Apple SSD without T2.

FYI, speeds do not seem very far off on other HFS+ HDDs and HFS+ SSDs that I just tested on, or far off from the screenshots posted on ADM's own website (http://www.katsurashareware.com/pgs/adm.html). Really think this is either the expected result in macOS via this benchmark, or there is a problem/issue with this particular benchmark test. These drives all behave appropriately on multiple machines. Seems like we're picking at straws with some of these benchmarks, especially when evaluating on a machine 7+ years old...

 

[thornslack]

It’s not the expected result given that multiple users have evidenced a decline in performance moving to apfs vs their prior performance in hfs+ using the same benchmarks.

If you wanted to back your claims you could run the same bench on an older os to see how it compares on your laptop.

But it doesn’t seem like nitpicking if apples new file system also comes with measurably reduced performance metrics. It would be interesting to see a wider set of testing results. You should forward this to barefeats perhaps for further testing?

 

[bsbeamer]

I have seven SSDs that are currently connected to MacPro5,1 that are all HFS+ formatted in High Sierra 10.13.6 (17G5019).

All are Samsung-based SSDs and non-Apple issued. Mixture of EVO 840 and EVO 850 right now. Two are in SATA sleds in RAID0 volume, two are in SATA sleds on their own volume, two are in Apricorn Velocity Duo X2 as own volume, and EVO 970 (NVMe via PX1 PCIe) as boot drive. Six volumes total with seven physical drives.

What tests do you want to see? They're all HFS+ formatted and showing nearly the same style results as MBP11,3 above that is APFS format.

 

[handheldgames]

I have seven SSDs that are currently connected to MacPro5,1 that are all HFS+ formatted in High Sierra 10.13.6 (17G5019).

All are Samsung-based SSDs and non-Apple issued. Mixture of EVO 840 and EVO 850 right now. Two are in SATA sleds in RAID0 volume, two are in SATA sleds on their own volume, two are in Apricorn Velocity Duo X2 as own volume, and EVO 970 (NVMe via PX1 PCIe) as boot drive. Six volumes total with seven physical drives.

What tests do you want to see? They're all HFS+ formatted and showing nearly the same style results as MBP11,3 above that is APFS format.

Try to add a partition one of the HFS+ SSD’s, format it APFS and re-run the benchmark to see the drop in performance.

 

[bsbeamer]

Six different tests performed under High Sierra 10.13.6 17G5019 via MacPro5,1 USB 3.0 (via PCIe) with SanDisk USB flash drive as base. The results are all similar regardless of the format. The only time you notice a significant write performance hit is with encryption. APFS Encrypted is/was the default for all SSD system drives that were converted to APFS format during High Sierra. Assume the same is true for all SSDs that were converted to APFS through Mojave installer - converted to APFS Encrypted vs APFS. Most SSD system drives were NOT formatted HFS+ Encrypted format previously when users formatted themselves. The majority formatted HFS+ (no encryption).

Will try to run tests of all system SSDs this evening.

FAT32:


ExFAT:


HFS+:


HFS+ Encrypted:


APFS:


APFS Encrypted:

 

[handheldgames]

Six different tests performed under High Sierra 10.13.6 17G5019 via MacPro5,1 USB 3.0 (via PCIe) with SanDisk USB flash drive as base. The results are all similar regardless of the format. The only time you notice a significant write performance hit is with encryption. APFS Encrypted is/was the default for all SSD system drives that were converted to APFS format during High Sierra. Assume the same is true for all SSDs that were converted to APFS through Mojave installer - converted to APFS Encrypted vs APFS. Most SSD system drives were NOT formatted HFS+ Encrypted format previously when users formatted themselves. The majority formatted HFS+ (no encryption).

Will try to run tests of all system SSDs this evening.

FAT32:
View attachment 819036

ExFAT:
View attachment 819037

HFS+:
View attachment 819038

HFS+ Encrypted:
View attachment 819039

APFS:
View attachment 819040

APFS Encrypted:
View attachment 819041

Close.... But off waaaay off target. Good to know flash drives are much less effected How about trying this on a SSD? I'd rather not play bring me a stone. It would also be helpful if you didn't crop the partition / test size from the screenshots. It helps to see how the tests are configured.
[doublepost=1548885826][/doublepost]Sharing another set of results - from a 2nd 186GB partition carved out of a 1tb 970Pro that's I'm running off of.

(left - APFS right- HFS)

HFS small 4k writes are about 50% faster yet, when accessing a separate M.2 slot.

 

[bsbeamer]

Results of all drives in system in High Sierra 10.13.6 17G5019 in MacPro5,1. All AJA System Test references are with 4GB test file source. All drives are populated and being used. Cannot be reconfigured with new formats or partitions at this time.

System drive - Angelbird Wings PX1 PCIe NVMe with EVO 970, HFS+:


Apricorn Velocity Duo X2 PCIe SATA, EVO 850 #1 (non-RAID), HFS+:


Apricorn Velocity Duo X2 PCIe SATA, EVO 850 #2 (non-RAID), HFS+:


SATA bay 1, EVO 840 #1 (non-RAID), HFS+:


SATA bay 2, EVO 840 #2 (non-RAID), HFS+:


SATA bays 3 & 4, EVO 840s (#3 & #4) in RAID0, HFS+:


USB 3, GRAID Hardware RAID0 (2xHDD), HFS+:


USB 3, HGST HDD via USB Drive Dock, HFS+:

 

[handheldgames]

Results of all drives in system in High Sierra 10.13.6 17G5019 in MacPro5,1. All AJA System Test references are with 4GB test file source. All drives are populated and being used. Cannot be reconfigured with new formats or partitions at this time.

System drive - Angelbird Wings PX1 PCIe NVMe with EVO 970, HFS+:
View attachment 819066

Apricorn Velocity Duo X2 PCIe SATA, EVO 850 #1 (non-RAID), HFS+:
View attachment 819064

Apricorn Velocity Duo X2 PCIe SATA, EVO 850 #2 (non-RAID), HFS+:
View attachment 819065

SATA bay 1, EVO 840 #1 (non-RAID), HFS+:
View attachment 819068

SATA bay 2, EVO 840 #2 (non-RAID), HFS+:
View attachment 819069

SATA bays 3 & 4, EVO 840s (#3 & #4) in RAID0, HFS+:
View attachment 819067

USB 3, GRAID Hardware RAID0 (2xHDD), HFS+:
View attachment 819070

USB 3, HGST HDD via USB Drive Dock, HFS+:
View attachment 819071

Thanks for sharing. The slowdown doesn't effect large file transfer. [ 4K ] and [ 4KQD32 ] writes seem to have the greatest impact. That would be the right column 2nd and 4th rows in the cropped screenshots, if you are using the same AmorphousDiskMark configuration as it's impossible to tell.

 

[bsbeamer]

This is all within range and expected behavior. Whacking a disk with tons of small files fills up a cache quickly and that is what you are observing. SSDs dump this more quickly than HDDs. NVMe dump these more quickly than SATA SSDs. You'll observe the same behavior on DSLR and pro video cameras, especially with RAW dumps to CF and other media. (When the buffer is full, you cannot continue shooting.) Dumping the cache while it's being repopulated is where your observing this benchmark slowdown. Again, believe this test is flawed in design and has not been updated in 5+ OS versions. You may be better off setting up a RAM disk if this is the type of real-world work you're doing.

Default settings on ADM used. I do not share anything with disk names because they are confidential client drives and do not risk that exposure.

 

[handheldgames]

This is all within range and expected behavior. Whacking a disk with tons of small files fills up a cache quickly and that is what you are observing. SSDs dump this more quickly than HDDs. NVMe dump these more quickly than SATA SSDs. You'll observe the same behavior on DSLR and pro video cameras, especially with RAW dumps to CF and other media. (When the buffer is full, you cannot continue shooting.) Dumping the cache while it's being repopulated is where your observing this benchmark slowdown. Again, believe this test is flawed in design and has not been updated in 5+ OS versions. You may be better off setting up a RAM disk if this is the type of real-world work you're doing.

Default settings on ADM used. I do not share anything with disk names because they are confidential client drives and do not risk that exposure.

Haha.. AmorphousDiskMark is not the issue.. APFS is. If you want to benchmark MacOS 4k performance, you have 2 choices: AmorphousDiskMark and Quickbench.

Anything else is focused on Large File Transfer. Alternatively, one can boot into Windows and test 4k performance with Crystal Disk Mark, which delivers results that are almost identical to AmorphousDiskMark. Hint: it's still relevant....

I generally run with a no virtual memory, 96 GB of ram and a 30 GB ramdisk. Some of my operations can't be offloaded to a ramdisk.

 

[bsbeamer]

Every single screen shot posted above is for HFS+ formatted SSDs, as stated. This is not an APFS issue if it also appears in HFS+.

 

[handheldgames]

Every single screen shot posted above is for HFS+ formatted SSDs, as stated. This is not an APFS issue if it also appears in HFS+.

Yes. Your HFS+ screenshots were well documented. However... The issue is seen when you take the same SATA II SSD, NVMe SSD, whole or partition, format it for APFS - benchmark with Amorphous, format it for HFS+ and benchmark. Simply showing benchmarks for HFS drives is 1/2 of the picture... Showing equivalent APFS benchmarks for each partitions out of reach in your instance, but is needed to complete the picture. It's like the yin.. missing the yang. Or the burger without the old lady yelling "Where's the beef!"

Any variance in [ 4k ] performance shown between devices is inherent of the drive / partition health. Once you have additional storage options, you should be able to reproduce the results.

Personally, I run with time machine and can be rather brutal with my storage. If space is available, I'll insert / delete / resize partitions as needed.

 

[bsbeamer]

Any variance in [ 4k ] performance shown between devices is inherent of the drive / partition health.

Your first post shows this:


How do explain the variance in SEQ QD32 write performance? Why is this not an issue? Just because it is substantially higher in APFS vs. HFS+?

At the end of the day, what do you want to happen as a resolution or fix to your issue? Apple to stop using the format? Apple to allow HFS+ for system drives in Mojave (or higher)? Truly not trying to get into an argument here. Just want to understand your expected or desired outcome.

My understanding was APFS was developed/designed/optimized/(whatever spin word they want to throw out) to take greater advantage of modern SSDs, controllers, etc. It APPEARS that your results do prove that to an extent, but your gripe is with the way it now handles smaller (4K) files.

Is the copy-on-write metadata feature of APFS the main culprit with smaller files in this test?
Does 32-bit in HFS+ vs 64-bit in APFS impact anything with this test?
Does UTF-8 vs UTF-16 play a factor?
Does the migration from 512 bytes per sector to 4K/4096 make a difference for this test?
Have you tested any drives/SSDs with 512 bytes per sector?

Some Intel SSDs can be changed/converted between the two physical sector sizes with firmware updates. Intel's SSD are here: https://www.intel.com/content/www/us/en/support/articles/000006392/memory-and-storage.html

Would drive controllers optimized by the manufacturer (or firmware changes/updates) to take full advantage of the way APFS works have an impact on this at all? Is it their responsibility to update their controllers/firmware, or do you believe this is all on Apple? Have any recent SSDs been made/released advertising APFS optimization?

Have you tested impacts with NTFS vs. ReFS at all?

 

[h9826790]

I bet that disk benchmark disabled cache intentionally in order to measure pure disk performance. However, that will be totally unrealistic.

e.g. When Apple design APFS, it should be use with cache (this is the normal situation). Therefore, the 4k no cache writing performance isn't that important. And they should focus on improving the real world whole system performance, but not pure disk 4k no cache writing performance.

I will say may be you really discovered one of the performance disadvantage by using APFS (vs HFS+). However, if that doses't matter in real world, is that really still so important?

 

[crjackson2134]

Is anyone else seeing these results?

Yes, I too have noticed this as well. Subjectively, I’ll add that HFS+ feels snappier when clicking on a target and waiting for a disk response. I won’t try to prove it to anyone, I have no need. It just feels that way to me.

I know, I’ll be the next person to be on the receiving end of contrary comments, but I do notice the same behaviors reported in your observations.