SSD + RAID 0 + Home Brew Fusion Drive (for fun and profit...)

[panjandrum]

Just received a massive donation of Mac Pros (4,1s, quickly becoming 5,1s) which allowed me to do an amusing test:

I created a RAID 0 array of 2x1TB WD Black drives, added a small 120GB SSD, and then created a Fusion drive out of the SSD + RAID array. (No worries about data-loss, it's just a test bed and it gets a nightly bootable CCC clone to single 2TB internal drive anyway).

I googled about a bit to see if anyone else had tried this, but couldn't really find anything.

I'll try to remember to stop back in a few months with a reliability report.

Next test? To see if slapping 2x2TB SSHDs into a Raid 0 and then making those part of an SSD+RAID 0 fusion drive will result in a massive Gnab Gib of data corruption... or not...

(Yes, I know it is silly and mostly pointless, I just want to see if it will work. So far, yes, it boots and functions as it should, but I have no idea if it will stay that way.)

 

[Blakehoo]

Just received a massive donation of Mac Pros (4,1s, quickly becoming 5,1s) which allowed me to do an amusing test:

I created a RAID 0 array of 2x1TB WD Black drives, added a small 120GB SSD, and then created a Fusion drive out of the SSD + RAID array. (No worries about data-loss, it's just a test bed and it gets a nightly bootable CCC clone to single 2TB internal drive anyway).

I googled about a bit to see if anyone else had tried this, but couldn't really find anything.

I'll try to remember to stop back in a few months with a reliability report.

Next test? To see if slapping 2x2TB SSHDs into a Raid 0 and then making those part of an SSD+RAID 0 fusion drive will result in a massive Gnab Gib of data corruption... or not...

(Yes, I know it is silly and mostly pointless, I just want to see if it will work. So far, yes, it boots and functions as it should, but I have no idea if it will stay that way.)

How'd you do it...?

 

[owbp]

@panjandrum , I assume you've made Recovery HD on SSD before merging it to Fusion with RAID array?
Can you boot into it (for the purpose of disabling SIP)?

I'm using 2x1TB RAID 0 for User "folder" and 120GB SSD for macOS, so I might try just that with my setup but I have to disable SIP for audio over HDMI.

 

[panjandrum]

@panjandrum , I assume you've made Recovery HD on SSD before merging it to Fusion with RAID array?
Can you boot into it (for the purpose of disabling SIP)?

I'm using 2x1TB RAID 0 for User "folder" and 120GB SSD for macOS, so I might try just that with my setup but I have to disable SIP for audio over HDMI.

I wouldn't suggest trying it unless it's a machine that's either backed-up routinely or doesn't have anything important on it. My guess it that Apple never thought anyone would be silly enough to try it and that things are going to go to heck soon... And, since I keep a backup drive in the system anyway, I have the recovery partition on that.

Blakehoo: Just in case you aren't aware of this, doing anything like creating RAIDs or Fusion drives will completely wipe all data from the drives in question... But assuming you know that, it is easy, just snag the instructions (I use the set by Other World Computing, they are pretty concise) but before you start the process to create the Fusion "drive" create a RAID 0 out of 2 or more other drives. When you get to this step:

diskutil coreStorage create FUSION /dev/disk1 /dev/disk2

enter the disk# of your RAID as the second "disk" (where it says "disk2" in the line above, enter the disk# assigned to your RAID, not the disk# assigned to any single disk in the system).

I used it through the day today and so far so good, with extremely good performance.

 

[Draeconis]

A guy on here asked me about creating a Fusion Drive RAID 1 setup, which I eventually got working.

It was bootable, and the drives were literally mirrored (SSD writing happened on the other SSD, etc).

I was skeptical that AppleRAID would accept logical volumes, but it did!

 

[eksu]

A guy on here asked me about creating a Fusion Drive RAID 1 setup, which I eventually got working.

It was bootable, and the drives were literally mirrored (SSD writing happened on the other SSD, etc).

I was skeptical that AppleRAID would accept logical volumes, but it did!

Can you share any more information on how you accomplished this? Very interested. Thanks.

 

[h9826790]

Can you share any more information on how you accomplished this? Very interested. Thanks.

I bet it's just make two Fusion Drive, and then RAID them together in disk utilise.

Apple software RAID almost accept any writable volume. I tried RAID a USB stick to a SSD, a drive to a partition, a SSHD to a HDD. It accept whatever combination I throw into it. Of course, the combination may degrade the performance or waste lots of space, or totally meaningless. But in my own test. Any combination work. I never use Fusion Drive. But I assume disk utility will treat it just like a single drive and accept it as a part of the RAID.

 

[eksu]

I bet it's just make two Fusion Drive, and then RAID them together in disk utilise.

Ideally I would like it to behave as if you raid the SSD's together, raid the HDD's together, then make a logical volume with those. I wonder if this gets around putting the recovery partition on raid.

Functionally, you'd want it to act like RAID 10, not RAID 01.

 

[goMac]

I googled about a bit to see if anyone else had tried this, but couldn't really find anything.

I had this configuration for a few years. One of the macOS updaters (I think El Capitan) flipped out when it saw that configuration. It might have been fixed since then, but I ended up moving back to a single HDD as part of the CoreStorage volume, and then eventually to a pure SSD setup.

 

[h9826790]

Ideally I would like it to behave as if you raid the SSD's together, raid the HDD's together, then make a logical volume with those. I wonder if this gets around putting the recovery partition on raid.

Functionally, you'd want it to act like RAID 10, not RAID 01.

My understanding is as follow.

Assuming you use the same SSD and HDD to create fusion drives. Due to the fusion drives are managed by the same OS, using the same logic, even RAID 1 after fusion, the data should still like SSD mirroring to SSD, HDD mirroring to HDD. Of course, the reason behind is very different.

But once one of the drive fail, one of the whole fusion drive will break, and when you replace the new drive, you must re-create the fusion, and let RAID 1 to handle the data sync. At that point, the data is no longer SSD mirroring to SSD, and HDD mirroring to HDD. But can only let OS to decide how to store data on the new fusion drive.

 

[Draeconis]

My understanding is as follow.

Assuming you use the same SSD and HDD to create fusion drives. Due to the fusion drives are managed by the same OS, using the same logic, even RAID 1 after fusion, the data should still like SSD mirroring to SSD, HDD mirroring to HDD. Of course, the reason behind is very different.

But once one of the drive fail, one of the whole fusion drive will break, and when you replace the new drive, you must re-create the fusion, and let RAID 1 to handle the data sync. At that point, the data is no longer SSD mirroring to SSD, and HDD mirroring to HDD. But can only let OS to decide how to store data on the new fusion drive.

So I created a test system with two SSDs and two HDDs. From this, you create two CoreStorage volumes from an SSD/HDD pair, which CoreStorage will identify as being able to utilise Fusion and work accordingly.

So, you'd have (roughly);
Physical Drives
disk0
disk1
disk2
disk3

Logical Drives (Fusion)
disk4
disk5

You'd then tell appleraid to mirror disk4 and disk5. Even though they're logical volumes spanning multiple disks, appleraid accepts this, and creates disk6, which is where your machine boots from.

This setup can suffer one disk failure, and boot perfectly fine, but beyond that the setup will break. If a disk drops out, its best to drop out of the mirror, rebuild the fusion disk from a spare, and re-mirror.

It has to be done this way; CoreStorage cannot create Fusion Drives from an appleraid volume, and you probably wouldn't want that anyway, since any single drive failure would take your system out completely.

It's a nice test, and certainly interesting to try out. In practice, I would assume macOS installers would throw a fit trying to do anything with it (especially with APFS in the near future). If it's something you want, then similar when a drive fails, if you wanted to upgrade macOS version; backup your system, break the mirror, upgrade, and re-create. Also, HFS+ is garbage, so creating something like this, while interesting, I wouldn't trust. Also, macOS doesn't warn you when a disk drops out, so if you're cognisant enough to create this in the first place, then you'd be wise to write something to monitor the health of your system.

Obviously, this is only simple when your SSD and HDD volumes match. The chap asking me about it had SSDs that were a different size. I recommended he partition the bigger SSD to match the smaller one, so as to simplify it. CoreStorage doesn't care what sizes the underlying disks are; it creates a logical volume spanning both volumes of the total size. You would mirror this to another CoreStorage logical volume of a different size, where the mirror will take the smaller size. However, if the underlying disks don't match, and mirroring seems to duplicate the write pattern, when Fusion Drive A writes something to the SSD that Fusion Drive B can't because it's smaller, then I don't know what happens.

What I haven't tried is appleraid mirroring two APFS logical volume fusion drives.. that would be more interesting, since APFS is (hopefully) far more resilient to failure than HFS+ is.

 

[h9826790]

So I created a test system with two SSDs and two HDDs. From this, you create two CoreStorage volumes from an SSD/HDD pair, which CoreStorage will identify as being able to utilise Fusion and work accordingly.

So, you'd have (roughly);
Physical Drives
disk0
disk1
disk2
disk3

Logical Drives (Fusion)
disk4
disk5

You'd then tell appleraid to mirror disk4 and disk5. Even though they're logical volumes spanning multiple disks, appleraid accepts this, and creates disk6, which is where your machine boots from.

This setup can suffer one disk failure, and boot perfectly fine, but beyond that the setup will break. If a disk drops out, its best to drop out of the mirror, rebuild the fusion disk from a spare, and re-mirror.

It has to be done this way; CoreStorage cannot create Fusion Drives from an appleraid volume, and you probably wouldn't want that anyway, since any single drive failure would take your system out completely.

It's a nice test, and certainly interesting to try out. In practice, I would assume macOS installers would throw a fit trying to do anything with it (especially with APFS in the near future). If it's something you want, then similar when a drive fails, if you wanted to upgrade macOS version; backup your system, break the mirror, upgrade, and re-create. Also, HFS+ is garbage, so creating something like this, while interesting, I wouldn't trust. Also, macOS doesn't warn you when a disk drops out, so if you're cognisant enough to create this in the first place, then you'd be wise to write something to monitor the health of your system.

Obviously, this is only simple when your SSD and HDD volumes match. The chap asking me about it had SSDs that were a different size. I recommended he partition the bigger SSD to match the smaller one, so as to simplify it. CoreStorage doesn't care what sizes the underlying disks are; it creates a logical volume spanning both volumes of the total size. You would mirror this to another CoreStorage logical volume of a different size, where the mirror will take the smaller size. However, if the underlying disks don't match, and mirroring seems to duplicate the write pattern, when Fusion Drive A writes something to the SSD that Fusion Drive B can't because it's smaller, then I don't know what happens.

What I haven't tried is appleraid mirroring two APFS logical volume fusion drives.. that would be more interesting, since APFS is (hopefully) far more resilient to failure than HFS+ is.

The whole setup should still allow 2 drives failure, however, must be within the same fusion drive. E.g.

Fusion drive A consist of SSD A + HDD A

Fusion drive B consist of SSD B + HDD B

Then RAID them together (RAID 1)

If SSD A fail, the FD A failed, but RAID 1 still works.

Of SSD + HDD A failed, FD 1 fail, RAID 1 should still work. Of course, after SSD A failed, HDD A should do nothing at all. Even it's still working, but same as failed in the data point of view.

But if SSD A + HDD B fail, then whole Fusion Drive failed and unable to recover data.

Therefore, if possible, form 2 SSD in a RAID 1. And another 2 HDD in RAID 1. And then fusion them together will have better redundancy. This allow the fusion drive continue to work in case SSD A + HDD B failed. However, AFAIK, we can't create fusion drive in this way.

For difference size SSD, I also believe that make a partition on the larger one to match the smaller one is the best way to go.

I am sure the RAID 1 can accept 2 fusion drive that formed by 4 different size drives. In this case, the performance should be ultimately limited by the HDD performance that on the fusion drive which has less SSD capacity. Just like RAID 1 a SSD with a HDD. We simply waste the SSD's performance, but the RAID 1 should work as expected.

 

[Draeconis]

The whole setup should still allow 2 drives failure, however, must be within the same fusion drive. E.g.

Fusion drive A consist of SSD A + HDD A

Fusion drive B consist of SSD B + HDD B

Then RAID them together (RAID 1)

If SSD A fail, the FD A failed, but RAID 1 still works.

Of SSD + HDD A failed, FD 1 fail, RAID 1 should still work. Of course, after SSD A failed, HDD A should do nothing at all. Even it's still working, but same as failed in the data point of view.

But if SSD A + HDD B fail, then whole Fusion Drive failed and unable to recover data.

Therefore, if possible, form 2 SSD in a RAID 1. And another 2 HDD in RAID 1. And then fusion them together will have better redundancy. This allow the fusion drive continue to work in case SSD A + HDD B failed. However, AFAIK, we can't create fusion drive in this way.

For difference size SSD, I also believe that make a partition on the larger one to match the smaller one is the best way to go.

I am sure the RAID 1 can accept 2 fusion drive that formed by 4 different size drives. In this case, the performance should be ultimately limited by the HDD performance that on the fusion drive which has less SSD capacity. Just like RAID 1 a SSD with a HDD. We simply waste the SSD's performance, but the RAID 1 should work as expected.

So, at a risk of repeating myself, you cannot create a CoreStorage logical volume from an AppleRAID volume. So what you propose is impossible.

 

[panjandrum]

So, at a risk of repeating myself, you cannot create a CoreStorage logical volume from an AppleRAID volume. So what you propose is impossible.

I did it the opposite way I think most people assumed I did: I built a single RAID 0 out of 2 drives FIRST, and then I created the FUSION (wait, do we need that to be all caps?) drive using a single small SSD and the logical volume created by the RAID setup. In theory this produces a FUSION which reads the SSD (not in RAID) plus the logical RAID 0 volume.

This worked without any errors during the FUSION drive setup procedure. However, we don't know about real-world reliability at this point. It definitely seems to be very quick; to see if I'm getting real RAID 0 speed once I hit the spinning drives I will go ahead and create a second internal RAID 0 out of 2 more drives and use those for CCC instead of the single drive I currently use. Since I have far more data on the FUSION+RAID drive than the SSD alone can handle, if I monitor drive throughput I should be able to see if I'm getting 2-drive RAID 0 performance (benchmarking software probably wouldn't do a good job due the the FUSION setup, but cloning 1.2TB will certainly give me a good idea what's going on). I won't have time to do this right away, but I'll try to remember to get to it in the next few weeks.

So here is an example setup of the way I did it:

Let's pretend I have only 3 drives, the 1x120GB SSD and 2x1TB HDD, and that when I run diskutil list they are listed as disk1, disk2 & disk3.

I create the RAID 0 out of the entire disks2 & disk3, which produces a logical volume RAID 0 array, let's say it is disk4.

The command to then create the FUSION drive would be:

diskutil coreStorage create FUSION /dev/disk1 /dev/disk4

I really didn't expect it to work, but it did, and is still working just fine. Zero problems since my OP and good performance.

 

[panjandrum]

I finally had time to stick 2 more drives in there and make them a second RAID 0 array. When copying sufficient data that I'm sure I'm not just hitting the SSD portion of my SSD+RAID 0 FUSION setup (for example, when using CCC to clone everything), I'm definitely getting RAID 0 speeds off the rotational drives. Still no problems at this point, which is a pleasant surprise considering I didn't even expect this to work, at all, in the first place. Disk Utility's First Aid feature consistently reports zero problems. So, I guess, if you use a FUSION system and want to maintain a bit more speed when you are moving data around on HDDs rather than the SSD portion, this might be a way to go. I still wouldn't recommend it unless you are as backup-paranoid as I am (which is extremely...)

 

[Blakehoo]

OK, Thank you for that!

 

[manHands]

@panjandrum

Any long term usage results? Are the HDDs set up as RAID0 a hardware or software raid?

 

[panjandrum]

No long-term problems that I encountered. I finally broke it apart and made a traditional BYO Fusion drive instead, mostly because I gained access to a 1TB SSD to make it with. I think I used my home-grown Fusion/Raid0 setup for about 18months. Something like that. All done using Apple's RAID utils (thus a software RAID).

 

[Kevtg]

No long-term problems that I encountered. I finally broke it apart and made a BYO Fusion drive instead, mostly because I gained access to a 1TB SSD to make it with. I think I used my home-grown Fusion/Raid0 setup for about 18months. Something like that. All done using Apple's RAID utils (thus a software RAID).

I have created a core storage volume thusly:
2x 4TB HDD in RAID 1 (AppleRAID)
1x 120GB SSD
This creates a single volume on my machine. This is not a boot drive.

My question is, how would I repair a drive or SSD failure? Is that even possible?

As an alternative, could I:
120GB SSD + 4TB HDD into a core storage volume
120GB SSD + 4TB HDD into a core storage volume

into a RAID 1

My goal is to have a cheap, fast, and redundant storage medium. This is not a boot drive. It backs up my boot drive and stores working data, videos, JPGs, etc. This would be periodically backed up to a single external drive.

 

[panjandrum]

I have created a core storage volume thusly:
2x 4TB HDD in RAID 1 (AppleRAID)
1x 120GB SSD
This creates a single volume on my machine. This is not a boot drive.

My question is, how would I repair a drive or SSD failure? Is that even possible?

As an alternative, could I:
120GB SSD + 4TB HDD into a core storage volume
120GB SSD + 4TB HDD into a core storage volume

into a RAID 1

My goal is to have a cheap, fast, and redundant storage medium. This is not a boot drive. It backs up my boot drive and stores working data, videos, JPGs, etc. This would be periodically backed up to a single external drive.

You can't repair individual drives in a Fusion configuration OR a RAID configuration. Or, at least, if you can't I sure don't know how! Never rely on configurations like these, or even the most basic traditional Fusion or Raid 0 arrays, unless you have it routinely backed-up (or just use it for crap data you don't need). Every drive you add is a point of failure.

 

[Kevtg]

You can't repair individual drives in a Fusion configuration OR a RAID configuration. Or, at least, if you can't I sure don't know how! Never rely on configurations like these, or even the most basic traditional Fusion or Raid 0 arrays, unless you have it routinely backed-up (or just use it for crap data you don't need). Every drive you add is a point of failure.

If a drive in a basic RAID 1 fails, you can replace the drive and rebuild the RAID. My question is, can I do the same in the event of a drive failure in any of my setups?

 

[panjandrum]

If a drive in a basic RAID 1 fails, you can replace the drive and rebuild the RAID. My question is, can I do the same in the event of a drive failure in any of my setups?

Sorry, should have said "RAID 0" configuration. I just now see that you have those as RAID 1 instead of RAID 0. My guess is that by doing a BYO Fusion drive that way you will negate any benefit of having those drives in RAID 1. I can't imagine you could rely on the ability to rebuild your RAID in a configuration where your RAID 1 array is part of a larger Fusion Array. As far as I know there is zero redundancy in even a basic Fusion drive setup, which seems to me like it's effectively a JBOD configuration with the added function that the OS knows that the first drive in the array is the SSD and the second drive is the HDD and then manages the data accordingly. I would definitely treat any of these configurations as "extremely fragile".

But there a certainly people here who know more about it that I do, so maybe someone else will tell you otherwise.

 

[porican]

Hey guys, i know this is an old thread, but I have been experimenting with a PCIe SSD + 2xHDD RAID 0 fusion setup, and I figured I'd report what I found.

The main takeaway? While it is possible to make an APFS fusion drive, for some reason Mojave does NOT like it when you try to create an APFS Fusion drive that includes a RAID 0 array. Every time I tried, whether from the initial Fusion setup in Terminal, or converting a completed HFS+ fusion setup to APFS after the fact, the cMP freaked out and immediately shut down. Converting the container back to HFS+ with disk utility in recovery mode seems to fix the problem.

I'm currently cloning my home folder to the newly created RAID 0 + PCIe SSD fusion set (with HFS+/CoreStorage), and will report back once it's up and running.

My setup, for the curious:

2010 cMP 5,1
2x3.06 6-core Xeon
Bootrom: 144.0.0.0.0
OS: macOS Mojave (10.14.6) on 256GB Samsung SM961 (slot 3)
Home folder:
Fusion drive: 2x2TB ironwolf HDD (SATA II bays 1 +2) + 128GB Apple OEM SSPOLARIS SSD (from 2019 iMac, Slot 2)
Home folder clone:
1x4TB HGST Ultrastar

 

[porican]

Hey guys, i know this is an old thread, but I have been experimenting with a PCIe SSD + 2xHDD RAID 0 fusion setup, and I figured I'd report what I found.

The main takeaway? While it is possible to make an APFS fusion drive, for some reason Mojave does NOT like it when you try to create an APFS Fusion drive that includes a RAID 0 array. Every time I tried, whether from the initial Fusion setup in Terminal, or converting a completed HFS+ fusion setup to APFS after the fact, the cMP freaked out and immediately shut down. Converting the container back to HFS+ with disk utility in recovery mode seems to fix the problem.

I'm currently cloning my home folder to the newly created RAID 0 + PCIe SSD fusion set (with HFS+/CoreStorage), and will report back once it's up and running.

My setup, for the curious:

2010 cMP 5,1
2x3.06 6-core Xeon
Bootrom: 144.0.0.0.0
OS: macOS Mojave (10.14.6) on 256GB Samsung SM961 (slot 3)
Home folder:
Fusion drive: 2x2TB ironwolf HDD (SATA II bays 1 +2) + 128GB Apple OEM SSPOLARIS SSD (from 2019 iMac, Slot 2)
Home folder clone:
1x4TB HGST Ultrastar

UPDATE: This has been up and running for about a week now, and seems to be working well. Blackmagic disk speed test results have plummeted (not sure why), but real-world usage definitely feels snappier. iTunes (with a 40,000+ song 630GB library) seems to be opening and closing much quicker (it used to get hung up a little on "saving iTunes library"), and my Lightroom library seems to load much quicker and response is snappier. Everything else seems to be responding just as quickly as before, so I think it's safe to call this a success!