MP 7,1 OpenZFS woes, on a maxed-out Mac Pro with Catalina 10.15.3

[mward333]

Is anybody out there using OpenZFS on a Mac Pro with Catalina 10.15.3 ??
I'm using OpenZFS 1.9.3.1 (https://openzfsonosx.org/) on a Mac Pro with 1.5 TB of RAM, 8 TB (Apple-installed) SSD, and 96 TB of spinning disks, namely: six of the 16 TB Seagate Exos drives (2 in a Pegasus J2i and 4 in a Pegasus R4i). The spinning disks are setup in one zpool, in a very basic way:
sudo zpool create -f -o ashift=12 tank disk2 disk3 disk4 disk5 disk6 disk7

I'm a longtime zfs user, but only used zfs (until now) on Linux servers. I'm running Catalina 10.15.3. I am able to import approximately 100 GB or sometimes as much as 200 GB at once, but if I do larger inputs than this, the data transfer will freeze, resulting in a need to reboot the Mac. I'm trying to move data from the Apple-installed 8 TB SSD (APFS format) to the zfs pool. The transfers are also much slower than we might expect, based on the published write speeds for these drives.

Not only the (big) writes are failing: (Big) reads are failing too. In other words, I am unable to transfer large amounts of data at once, from the zfs pool back to the (APFS format) SSD.

I've spent the last week (very frustrated) about this, after countless freezing and rebooting of the Mac Pro.
I tried to post on the OpenZFS forum but did not (yet) get a solution, and I have not heard from the experiences of others who are trying to use OpenZFS on either of their Pegasus enclosures inside the new Mac Pro.

All suggestions are welcome. As I wrote this, I tried to move 533 GB from the APFS SSD to the zpool, and it died after 82.69 GB of data was transferred.

P.S. My data consists of (roughly) 10 GB per file, plain ASCII text, nothing strange here. Thanks for listening!
P.P.S. I have even tried this many times after shutting down the Mac and starting things up fresh, so that the Mac is completely cool when I am attempting such data transfers, just to make sure that I don't have overheating CPU or overheating drives. The failures occur within the first 60 to 90 seconds of the (large) data writes or reads.
[automerge]1581122504[/automerge]
I'm not a newbie with regard to large data projects. A couple years ago, I ran a job on our clusters at our university that rendered 72 petabytes of data, and used 37 years of computing time on the cluster (of course, the computational tasks occurred in parallel).

 

[ZombiePhysicist]

Let's start with some basic tests. Copy the data from the 8TB internal Apple drive to some other spot on the internal drive? Any problems? Then, try copying the data to some known plugged in external USB drive. Does it work?

Then, nuke the OpenZFS drives and format them APFS as individual drives. Does copying work to all of them individually? Then nuke them, and RAID them. Does copying still work if it's APFS formatted?

If all the above works, and I suspect it will, my guess is there is something wonky with the OpenZFS driver, and it's setup. But I'd try to confirm that all the hardware is solid in and of itself.

ALSO, have you done some basic ram tests? You have a LOT of ram, and one bum stick could ruin your day. I'd do at least a quick ram test.

Kickbutt setup, hoping it works out for you!

 

[AidenShaw]

Let's start with some basic tests. Copy the data from the 8TB internal Apple drive to some other spot on the internal drive? Any problems? Then, try copying the data to some known plugged in external USB drive. Does it work?

Then, nuke the OpenZFS drives and format them APFS as individual drives. Does copying work to all of them individually? Then nuke them, and RAID them. Does copying still work if it's APFS formatted?

If all the above works, and I suspect it will, my guess is there is something wonky with the OpenZFS driver, and it's setup. But I'd try to confirm that all the hardware is solid in and of itself.

ALSO, have you done some basic ram tests? You have a LOT of ram, and one bum stick could ruin your day. I'd do at least a quick ram test.

Kickbutt setup, hoping it works out for you!

Good advice, but I'd also run S.M.A.R.T. checks in 'disk utility' or your favorite third party disk tool.

You might find a disk with problems much faster.

 

[Snow Tiger]

Good advice, but I'd also run S.M.A.R.T. checks in 'disk utility' or your favorite third party disk tool.

You might find a disk with problems much faster.

Concur here most heartily .

I can find some pretty nifty almost new enterprise mechanicals on eBay . But there's a reason why they are so cheap . They don't entirely pass their SMART tests and they get mustered out of duty once Mr. Data Center figures it out .

The trick is to find out what is not working as well as it should and whether you can live with it .

There's an excellent , perhaps the very best in the macOS community , SMART utility called - wait for it - SMART Utility by Volitans Software .

SMART Utility | Volitans Software

www.volitans-software.com

Makes Apple's Disc Utility's SMART function seem like it's Amateur Day .

 

[mward333]

There's an excellent , perhaps the very best in the macOS community , SMART utility called - wait for it - SMART Utility by Volitans Software .

Thanks for the suggestion, @Snow Tiger, about Volitans Software SMART Utility.
Looks like it is not available for folks who have Catalina.
It says it still might work if it was installed *before* Catalina, but the Mac Pro comes with Catalina, so I am doubtful that it is possible to go back to an earlier OS.
Back in June 2019, Volitans thought it was going to work with Catalina:

https://www.volitans-software.com/2019/06/smart-utility-and-mac-os-x-catalina-10-15/

but then in October 2019, they decided it will not (yet) work with Catalina:

https://www.volitans-software.com/2019/10/smart-utility-and-mac-os-x-catalina-10-15-update/

and if you go to their main webpage to buy the software:

SMART Utility | Volitans Software

www.volitans-software.com

you will find these notes. The link to purchase the software clearly says that it only works through Mojave.

Nonetheless, I appreciate your suggestion!
[automerge]1581164444[/automerge]

Good advice, but I'd also run S.M.A.R.T. checks in 'disk utility' or your favorite third party disk tool.

@AidenShaw, I always appreciate your advice. Unfortunately S.M.A.R.T. status is "Not Supported" on these drives.

 

[mward333]

Then, nuke the OpenZFS drives and format them APFS as individual drives. Does copying work to all of them individually? Then nuke them, and RAID them. Does copying still work if it's APFS formatted?

@ZombiePhysicist, thank you for your advice! OK, I nuked the OpenZFS drives and put them all into one large (96 TB) RAID0 array with APFS formatting, and I was able to copy more than 500 GB of data from the 8 GB SSD (also APFS formatted) to this RAID0 array in about 7 minutes, with no problems or hiccups at all. Fast and easy. This is how things should work in practice!

If all the above works, and I suspect it will, my guess is there is something wonky with the OpenZFS driver, and it's setup.

I'm really becoming convinced that there is some kind of incompatibility between the OpenZFS driver and the move from Kernel Extensions to System Extensions in Catalina. This has to be the culprit. I'm putting my money on this being the problem.

ALSO, have you done some basic ram tests? You have a LOT of ram, and one bum stick could ruin your day. I'd do at least a quick ram test.

I haven't done RAM tests yet, but I'm open to trying such tests. What do you recommend? Just the basic Apple Hardware Test?

 

[mward333]

Looks like some other users are having kernel panics with high I/O when using OpenZFS on Catalina. This sure sounds like the problems I am having. I hope for a resolution. I posted some information about this on one of their GitHub threads:

macOS 10.15 Catalina support ✅ · Issue #721 · openzfsonosx/zfs

It's still early, so I don't expect there to already be support for the new macOS Catalina beta, but surprisingly it worked! I figured I'd open a ticket to help track progress on any bugs. (Also to...

github.com

I would love to get this resolved. I'm working on a very data-intensive research problem, and I want to get back to work on it a.s.a.p.

 

[Snow Tiger]

Thanks for the suggestion, @Snow Tiger, about Volitans Software SMART Utility.
Looks like it is not available for folks who have Catalina.
It says it still might work if it was installed *before* Catalina, but the Mac Pro comes with Catalina, so I am doubtful that it is possible to go back to an earlier OS.
Back in June 2019, Volitans thought it was going to work with Catalina:

I think I figured out a method of getting SMART Utility by Volitans to work on a MP7,1 .

Obtain an external drive , load Mojave onto it using a Mac that natively supports this OS , install Volitan's SU , buy and update this app , update the drive to Catalina . Remove the external drive from the first Mac and then boot this drive with your MP7,1 . Make certain you allow booting from external media in the Startup Security Utility of your Mac Pro 7,1 .

I read the dev notes from Volitan on this issue . The program runs in Catalina , but must first be installed in some earlier macOS version .

 

[MacUser2525]

Looks like some other users are having kernel panics with high I/O when using OpenZFS on Catalina. This sure sounds like the problems I am having. I hope for a resolution. I posted some information about this on one of their GitHub threads:

macOS 10.15 Catalina support ✅ · Issue #721 · openzfsonosx/zfs

It's still early, so I don't expect there to already be support for the new macOS Catalina beta, but surprisingly it worked! I figured I'd open a ticket to help track progress on any bugs. (Also to...

github.com

I would love to get this resolved. I'm working on a very data-intensive research problem, and I want to get back to work on it a.s.a.p.

You have your solution to do that, use the RAID. Though not the 0 variety unless you want no data backup involved. And I doubt this is why you choose to try the ZFS for a no redundancy strategy. You let perfection get in the way of good enough to do the job now. Not that ZFS is perfect, their claims for it are a little weak from my experience with a drive failure in a redundant array. I had to wipe it and start from new when putting in identical replacement drive it would not do the resilver I tried no matter what. Luckily I have a triple backup strategy for my data on three different machines and sets of drives.

 

[mward333]

You have your solution to do that, use the RAID.

@MacUser2525 thanks for chiming in.... but I do want OpenZFS. Here's why. I would normally set it up this way:
sudo zpool create -f -o ashift=12 -O compression=lz4 -O casesensitivity=insensitive -O atime=off -O normalization=formD tank raidz2 disk2 disk3 disk4 disk5 disk6 disk7

I really want the compression that (Open)ZFS provides because the data generated by my research problem is easily compressed. Indeed, I am getting roughly 8x compression when I use zfs. I am very familiar with using zfs on our Linux servers. RAID on its own does not provide the (automatic) compression that I need. I was only describing a very simple OpenZFS configuration because my Mac Pro continues to die in any OpenZFS configuration (e.g., with or without compression), and I'm trying to narrow things down as much as possible, to identify the culprit here!

 

[Snow Tiger]

@MacUser2525 thanks for chiming in.... but I do want OpenZFS. Here's why. I would normally set it up this way:
sudo zpool create -f -o ashift=12 -O compression=lz4 -O casesensitivity=insensitive -O atime=off -O normalization=formD tank raidz2 disk2 disk3 disk4 disk5 disk6 disk7

I really want the compression that (Open)ZFS provides because the data generated by my research problem is easily compressed. Indeed, I am getting roughly 8x compression when I use zfs. I am very familiar with using zfs on our Linux servers. RAID on its own does not provide the (automatic) compression that I need. I was only describing a very simple OpenZFS configuration because my Mac Pro continues to die in any OpenZFS configuration (e.g., with or without compression), and I'm trying to narrow things down as much as possible, to identify the culprit here!

A bit off topic - I'm curious to know what memory modules you installed to get your 1.5TB ?

 

[mward333]

A bit off topic - I'm curious to know what memory modules you installed to get your 1.5TB ?

@Snow Tiger, I ended up buying 12 of these:

M386AAG40MMB-CVF - Samsung 1x 128GB DDR4-2933 LRDIMM PC4-23466U-L Quad Rank x4 Module

Get ➡️ Lifetime warranty ✅ Great savings ✅ Buy factory original M386AAG40MMB-CVF - Samsung 1x 128GB DDR4-2933 LRDIMM PC4-23466U-L Quad Rank x4 Modules @Memory.NET!

memory.net

I called Apple to verify that these should work. I also found this post by @repoman27 to be very helpful:

Ram: Samsung vs Crucial

I already posted this in another thread on Mac Pro (2019) RAM upgrades, but figured I'd drop it here as well. If you're not savvy with upgrading memory on Intel's recent server platforms, there's a lot to digest. Apple's pages on memory specifications and installing or replacing memory in the...

forums.macrumors.com

Also, be wary of some websites selling RAM that is compatible with a certain brand/type but not clearly stating what type you will receive. I liked the fact that this is a reputable seller (with a very long history) and they make it very obvious to see what specs and which specific manufacturer you will get.
Good luck if you buy something like this! My RAM arrived very quickly and, aside from this zfs trouble that I'm having, it seems to be working great!

 

[Snow Tiger]

@Snow Tiger, I ended up buying 12 of these:

M386AAG40MMB-CVF - Samsung 1x 128GB DDR4-2933 LRDIMM PC4-23466U-L Quad Rank x4 Module

Get ➡️ Lifetime warranty ✅ Great savings ✅ Buy factory original M386AAG40MMB-CVF - Samsung 1x 128GB DDR4-2933 LRDIMM PC4-23466U-L Quad Rank x4 Modules @Memory.NET!

memory.net

I called Apple to verify that these should work. I also found this post by @repoman27 to be very helpful:

Ram: Samsung vs Crucial

I already posted this in another thread on Mac Pro (2019) RAM upgrades, but figured I'd drop it here as well. If you're not savvy with upgrading memory on Intel's recent server platforms, there's a lot to digest. Apple's pages on memory specifications and installing or replacing memory in the...

forums.macrumors.com

Also, be wary of some websites selling RAM that is compatible with a certain brand/type but not clearly stating what type you will receive. I liked the fact that this is a reputable seller (with a very long history) and they make it very obvious to see what specs and which specific manufacturer you will get.
Good luck if you buy something like this! My RAM arrived very carefully and, aside from this zfs trouble that I'm having, it seems to be working great!

$1300 each . That's not too bad a price for a Sammy 128GB module . And you got a lifetime warranty , too . For the generic brand modules , it was until recently possible to grab the 128GB modules for around a thousand each . No more .

Lately , they've jacked up the price as if too many ppl were asking the retailers the same question : "is this compatible with my Mac Pro ?"

"Why , ye$ , $ir , now that you mention it ! How many of the$e would you like ?"

It's kind of predictable , I guess . We always are expected to pay more due to Apple's reputation for asking top dollar for every ... little ... thing .

 

[mward333]

$1300 each . That's not too bad a price for a Sammy 128GB module . And you got a lifetime warranty , too .
....
We always are expected to pay more due to Apple's reputation for asking top dollar for every ... little ... thing .

This was definitely less expensive than buying the RAM directly from Apple, and I feel very good about Samsung RAM. I also felt good about memory.net. I never bought from them in the past, but I checked them out very extensively, and they have been around for a *long* time in this business. So I felt reassured!

 

[Snow Tiger]

This was definitely less expensive than buying the RAM directly from Apple, and I feel very good about Samsung RAM. I also felt good about memory.net. I never bought from them in the past, but I checked them out very extensively, and they have been around for a *long* time in this business. So I felt reassured!

Sammy usually makes very good products .

Just don't drop a Galaxy Note 7 into a Samsung washing machine . If you do , run for the hills quick !

I greatly preferred to use Samsung enterprise DDR3 modules in all my MP4,1 and 5,1 builds . I don't believe a single one failed . Had some Hynix , Micron and Nanya component based modules fail , but memory failures are getting rarer now . Failures were common with DDR2 , as they often overheated . But DDR3 and DDR4 are much more reliable technology .

 

[AidenShaw]

I always appreciate your advice. Unfortunately S.M.A.R.T. status is "Not Supported" on these drives.

Are your sure that it's not just disabled? 12 TB Exos:

smartctl 7.0 2018-12-30 r4883 [x86_64-w64-mingw32-w10-1809] (sf-7.0-1)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     ST12000NM0008-2H3101
Serial Number:    ZHZ
LU WWN Device Id: 5 000c50
Firmware Version: SN02
User Capacity:    12,000,138,625,024 bytes [12.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sun Dec 01 10:22:55 2019 PST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.

 

[mward333]

@AidenShaw how do I find out? Or more appropriate for me to ask: How can I enable the S.M.A.R.T. status?

 

[mward333]

A user named rottegift is giving me a full analysis over at:

macOS 10.15 Catalina support ✅ · Issue #721 · openzfsonosx/zfs

It's still early, so I don't expect there to already be support for the new macOS Catalina beta, but surprisingly it worked! I figured I'd open a ticket to help track progress on any bugs. (Also to...

github.com

if anyone is interested! Wow!

 

[ZombiePhysicist]

@ZombiePhysicist, thank you for your advice! OK, I nuked the OpenZFS drives and put them all into one large (96 TB) RAID0 array with APFS formatting, and I was able to copy more than 500 GB of data from the 8 GB SSD (also APFS formatted) to this RAID0 array in about 7 minutes, with no problems or hiccups at all. Fast and easy. This is how things should work in practice!



I'm really becoming convinced that there is some kind of incompatibility between the OpenZFS driver and the move from Kernel Extensions to System Extensions in Catalina. This has to be the culprit. I'm putting my money on this being the problem.



I haven't done RAM tests yet, but I'm open to trying such tests. What do you recommend? Just the basic Apple Hardware Test?

Ok since the basic drives seem to be working it looks like we narrowed it down to the drivers, and personally, that’s what I would concentrate on.

It is weird you don’t see smart status on the exos drives. I have them and they show status.

I might try this:

Techtool Pro 20

Techtool Pro is Mac utility software for maintaining and troubleshooting your Mac, including drive repair, RAM testing, and more.

www.micromat.com

It has some smart status, ram tests, and works with Catalina.

I wonder if the Pegasus controller is somehow blocking the smart status, but I also have a j2i with a 16tb exos in it and the smart status shows up. I’m not sure why it wouldnt show up for you. At least the ones plugged in straight to the motherboard sata plugs. Something doesn’t add up there...

I wonder if some remnants of the zfs kernel extension block the status. I loathe installing any extensions other than standard Apple ones as they almost always cause trouble. Be sure you nuked all remnants of the ZFS drivers. Boot off a clean install of Catalina (Maybe on an external scratch drive ssd) and see if the status comes up under a clean boot.

 

[mward333]

Ok since the basic drives seem to be working it looks like we narrowed it down to the drivers, and personally, that’s what I would concentrate on.

It is weird you don’t see smart status on the exos drives. I have them and they show status.

I might try this:

Techtool Pro 20

Techtool Pro is Mac utility software for maintaining and troubleshooting your Mac, including drive repair, RAM testing, and more.

www.micromat.com

It has some smart status, ram tests, and works with Catalina.

I wonder if the Pegasus controller is somehow blocking the smart status, but I also have a j2i with a 16tb exos in it and the smart status shows up. I’m not sure why it wouldnt show up for. At least the ones plugged in straight to the motherboard sata plugs. Something doesn’t add up there...

I wonder if some remnants of the kernel extension of ads block the status. I loathe installing any extensions other than standard Apple ones as they almost always cause trouble. Be sure you nuked all remnants of the ZFS drivers. Boot off a clean install of Catalina (Maybe on an external scratch drive ssd) and see if the status comes up under a clean boot.

@ZombiePhysicist thank you for the suggestions! Indeed, it is strange that I don't see the S.M.A.R.T. status on the 16 TB Exos in the J2i and you do see it. Strange!

Anyway, like I said, I am getting some serious advice at over on the OpenZFS on X GitHub:

macOS 10.15 Catalina support ✅ · Issue #721 · openzfsonosx/zfs

It's still early, so I don't expect there to already be support for the new macOS Catalina beta, but surprisingly it worked! I figured I'd open a ticket to help track progress on any bugs. (Also to...

github.com

and we have a hypothesis that SPOTLIGHT is what is hurting me after all. Wow. I hate how Spotlight can tinker with other processes like this sometimes. We're working on what will hopefully be a resolution!

I'll try and boot later off a fresh Catalina installation, to see if I see the OpenZFS installation is what is blocking the S.M.A.R.T. status!

 

[ZombiePhysicist]

@ZombiePhysicist thank you for the suggestions! Indeed, it is strange that I don't see the S.M.A.R.T. status on the 16 TB Exos in the J2i and you do see it. Strange!

Anyway, like I said, I am getting some serious advice at over on the OpenZFS on X GitHub:

macOS 10.15 Catalina support ✅ · Issue #721 · openzfsonosx/zfs

It's still early, so I don't expect there to already be support for the new macOS Catalina beta, but surprisingly it worked! I figured I'd open a ticket to help track progress on any bugs. (Also to...

github.com

and we have a hypothesis that SPOTLIGHT is what is hurting me after all. Wow. I hate how Spotlight can tinker with other processes like this sometimes. We're working on what will hopefully be a resolution!

I'll try and boot later off a fresh Catalina installation, to see if I see the OpenZFS installation is what is blocking the S.M.A.R.T. status!

Again, make a clean install on an external and boot it. Before you go layering beta kernel extensions atop of something already not working. See if you can see the SMART status off a clean install, at least off the J2i. You SHOULD see it. If not, I suggest you have other problems with that system, and layering beta drivers over the top of something already not working right, it's a recipe for further problems down the road.

You need to gain confidence that the underlying hardware is right and working clean, IMO, before doing the other stuff.

Good luck with it!

 

[edgerider]

my two cent :
after trying almost evry thunderbolt /usb3 / software/ nas existing, I came to a very basic conclusion : for peace of mind/cost/ease of use nothing beats an CMp 5.1 with a pcie ssd, a good atto fastrack nt12 and one areca 1882x with netapp ds4346 disk shelfs in your basement.

once you try the peace of mind of true areca hardware raid with everything redondant and with a hotspare, you will never go back.
i setup my 24 drive in 3 pool of 8 drive in raid 60, or i do a 22 disk raid 6 with 2 hotspare, and it is fantastic....

netapp ds4346 can be bought dirt cheap because everybody think it is proprietary because it use stupid qsfp+ connector, but you just need a cable that is qsfp+ to sff8088, and the areca raid card see it as an expender.

you can find ds4346 24x3 or 24x4 for less than a grant on ebay.
just make sure you get 3/4 extra drives when you buy it.

everything is redondant on those diskshelves, and they are almost indestructible...

best part is it is sas3/sata6g and got a passthru port ... so basically if you need more space you just buy another diskshelf , chain it on the first one, and voila, you expend your raid volume.

there is some really cool features, like hdd cloning , alarm, automatic hotspare, etc etc...

but the raid is on the drives, not on tve card, so if your raid card burns, just replace it and you are back in business!

arreca 1882 can be buy for 150 to 200$ used of ebay... i alway have two spare ready to go.

and you can pilot the raid card over the network.

with my xserve and two 10gbe link , i got 1.2 gb/s R/W via link agrégation.
if you have a spare 2000$ you can buy 2 40Gb fastframe card of ebay, and transfer up to 2gb/s...

never had anything snapier and more reliable ....

I also have two ds4346 disk shelf with an entire back up of my most important stuff that is stored at a friends place... unplugged.

when you need it you just need to plug it and the raid is there...

the only downside is power consumption and stupid loud fans....
no seriously it makes the noise of 10 hair dryer....

 

[mward333]

@edgerider thanks for chiming in! I'm imagining your setup in my mind's eye now! This is awesome.
I'm a professor at a major research university, so I don't need to put shelves in my basement with racks of storage. We are fortunate to have crazy amounts of storage on our computing clusters at work.
For example, we just added another petabyte of storage for the undergraduate students in my class this summer. Seriously. I can't make this stuff up.

 

[deconstruct60]

...
Anyway, like I said, I am getting some serious advice at over on the OpenZFS on X GitHub:

macOS 10.15 Catalina support ✅ · Issue #721 · openzfsonosx/zfs

It's still early, so I don't expect there to already be support for the new macOS Catalina beta, but surprisingly it worked! I figured I'd open a ticket to help track progress on any bugs. (Also to...

github.com

and we have a hypothesis that SPOTLIGHT is what is hurting me after all. Wow. I hate how Spotlight can tinker with other processes like this sometimes. We're working on what will hopefully be a resolution!
....

Spotlight probably wasn't helping. However, this 1.5TB RAM configuration is probably tested on extremely little software. I highly doubt Apple has been running around helping super low budget open source projects figure out if they have any scaling issues. (the thread is pointing finger at 26GB of disk cache at this point. ).


Max Mac memory not to long ago maxed out in the 128GB range. Jumped to 256GB to 512TB and now into big dollar budget 1.5TB range. Pretty close to an order of magnitude. ZFS can be a bit of a RAM hog but just gobbling more and more doesn't always work well.

 

[mward333]

@deconstruct60, I agree with you. This is a big jump in the amount of available RAM. We use RAM in these quantities all the time on our Linux servers in my university, but it is (as you said) a big jump for Apple's Mac Pro. I hope that we can fine-tune the parameters so that OpenZFS will work for me. I want to do some high I/O from the Apple-installed SSD to/from the OpenZFS pool, but the reading/writing keeps getting throttled. Almost every single time.... Argh......