MP 1,1-5,1 5,1 + Cubix Xpander - Ideal PCIe Card Layout?

[Grumply]

Hi guys,

I’ve just ordered the final parts for my new (and fairly ludicrous) 5,1 build, which will include a Cubic Desktop Xpander (which expands one PCIe x16 slot into four). And I’m wanting to pick people’s brains on the ideal layout of the PCIe cards in order to maximise bandwidth across the lanes when working in Davinci Resolve.


The full build will include the following parts:

• 12-core 3.46Ghz CPU
• 96GB 1333Mhz RAM
• 2x Radeon VII GPUs
• Media Drive: 6x 6TB Seagate Ironwolf 3.5” SATA Drives in RAID 10 (internally)
• Boot Drive: 512GB Samsung 970 Pro NVME (on 4x PCIe 3.0 card)
• Cache Drive: 2x 2TB Samsung 960 Pro NVME RAID 0 (on 8x PCIe 3.0 Card)
• Blackmagic Decklink Mini Monitor 4k
• Sonnet Allegro Pro USB 3.1 4-port USB Type-A Card

My primary objectives would be to:

1. Maximise the processing throughput of the two Radeon VIIs
2. Maximise write speeds to the 2x 2TB NVME Cache Drive
3. Maximise the overall speed of the 512GB NVME Boot Drive

Basically this amounts to whether it’s better to keep the Cache Drive internally on a x16 Slot to maximise write speed and whether that would cost much GPU performance (having both GPUs in the Xpander) over keeping one of the Radeon VIIs internally on Slot 1?

The USB3.1 card will basically only be used to transfer data to the internal 6x 3.5” SATA RAID, so that shouldn’t be using up any bandwidth when I’m actually working (which makes me think it’s a good candidate for living in the Cubix).

I don’t know how much bandwidth the Boot Drive is likely to be using when I’m working (since all of the Media will be read from the 6x 3.5” SATA RAID, and all of the writing will happen to the NVME RAID Cache Drive. But I’m guessing it could potentially affect throughput to the Cache Drive or GPUs (if they’re in the Xpander at the same time). But would having it feed into the x16 slot (via the Xpander) allow it to run at higher speeds (compared to its 1500MB/s limit in the internal x4 slots) even with the other demanding components in the Cubix at the same time?

I’d appreciate any thoughts or suggestions anyone could offer, as expanding PCIe slots is a new game to me.

Here are the four basic layouts I’m currently pondering the efficacy of:

Cheers

 

[startergo]

I’ve just ordered the final parts for my new (and fairly ludicrous) 5,1 build, which will include a Cubic Desktop Xpander

Just out of curiosity if you are willing to spend around 4K for the Cubix expander why did you not consider buying the new MacPro? It would be a better investment.

 

[Grumply]

Just out of curiosity if you are willing to spend around 4K for the Cubix expander why did you not consider buying the new MacPro? It would be a better investment.

Well, after over a month of umming and ahhing over the ridiculous pricing, I actually had a 7,1 purchase configured and ready to order... and then got an offer of only 25% of the price covered under the 12 months interest-free financing deal they were offering through a subsidiary - which put a stop to the whole thing. ?

I then came very close to building a Threadripper 3 beast instead (which I'd been tossing up alongside the possibility of purchasing the 7,1) and leaving Mac behind entirely. But ultimately decided I didn't want to deal with a few of the intrinsic workflow limitations of moving to Windows with the software and codecs I use (it would have been a $5000 saving over the 7,1 but I was still looking at a $9000 machine, so it seemed silly to compromise at that price point).

All of which led me back to buying another 5,1 (after having just sold my previous one to help fund the 7,1). I already had the Radeon VIIs and the 4TB of NVME drives (which had been bought pre-emptively for the 7,1) so it became a question of how to stuff everything into a 5,1 with only 4 PCIe slots.

That configuration dilemma turned into a month of spreadsheeting and pulling my hair out, and then someone came to my rescue with an offer on a used Cubix that I couldn't refuse. That's what made the whole thing possible. I could never have justified the retail pricing for a 5,1 build.

I'd actually been looking at repurposing an old Nvidia Quadro Plex GPU box, or buying a Netstor expander box, or building my own x16 PCIe expansion backplate... alongside drilling holes in the side of the Mac Pro to feed an external PSU inside to the two Radeon VIIs, and running the two NVME drives in idividual caddies into the UCB3.1 card (and then RAID 0ing them across the card's two controllers, to get a fast cache drive while still being able to fit the Decklink card and the two Radeon VIIs inside). And you don't want to know the depths of complexity I've gone to trying to figure out internal media storage! With the second GPU blocking three of the internal drive bays, I'd have had to physically strip the boot SSD of it's casing, and mount it self-suspended in the 2nd drive bay to get away with it all.

So it's been a journey! But the Cubix takes most of the stress out of things now. I just have to get it configured as best as possible.

 

[tsialex]

You will need to make some real life tests to see what will improve your workflow the best, since you don't have enough lanes for everything you wanna connect and something always will have to be shared. Maybe both GPUs inside the chassis will work best for your storage throughput, since the GPUs will work fine with the PCIe switch inside the Cubix Expander.

 

[bsbeamer]

When clients were using Cubix Xpanders for multiple GPU setups in the past, it was ALWAYS an issue if the native GPU was not present in the double height Slot #1. That being said, I'd highly consider expanding using Slot #2 for multiple GPUs along with your main GPU (for driving monitor/monitors) in Slot #1.

It worked well with numerous NVIDIA cards back before Sierra/High Sierra, but became more trouble than it was worth around Sierra.

 

[Grumply]

When clients were using Cubix Xpanders for multiple GPU setups in the past, it was ALWAYS an issue if the native GPU was not present in the double height Slot #1. That being said, I'd highly consider expanding using Slot #2 for multiple GPUs along with your main GPU (for driving monitor/monitors) in Slot #1.

It worked well with numerous NVIDIA cards back before Sierra/High Sierra, but became more trouble than it was worth around Sierra.

Good to know, thanks ?I'll start with one of the Radeon VIIs in Slot 1, and then see how the two NVME drive setups perform both running alongside each other in the Cubix, and with the Boot Drive mounted internally.

Do you know if there are any issues having the boot drive installed in the Cubix?

 

[bsbeamer]

Do you know if there are any issues having the boot drive installed in the Cubix?

Have not been on a Cubix setup since the bootable NVMe firmware was released, so cannot comment with certainty. Most were bootable SATA RAID setups (pre Sierra) or SATA SSD (direct or via PCIe adapter).

The Cubix was really great for awhile for very specific needs, but make sure they are still offering support for MP5,1 in some capacity. Two client offices basically liquidated their Cubix hardware 4-5 years ago. One office kept a unit around as a just in case and it really acted up in Mojave testing. There are many "it just works" stuff involved with the box, but there were additional software tools years ago.

 

[Grumply]

Have not been on a Cubix setup since the bootable NVMe firmware was released, so cannot comment with certainty. Most were bootable SATA RAID setups (pre Sierra) or SATA SSD (direct or via PCIe adapter).

The Cubix was really great for awhile for very specific needs, but make sure they are still offering support for MP5,1 in some capacity. Two client offices basically liquidated their Cubix hardware 4-5 years ago. One office kept a unit around as a just in case and it really acted up in Mojave testing. There are many "it just works" stuff involved with the box, but there were additional software tools years ago.

???

Fingers firmly crossed. From the little info I've been able to gather online, it sounds like most of the issues people have had are related to using Nvidia GPUs with the Cubix and 5,1s - and all of the fresh hell, that driver limitations and incompatible OS versions have caused with that.

I'm hoping that the Radeon VIIs being supported natively in Mojave 10.14.6 (the single internally-mounted one I'm running at the moment, is running beautifully), will allow the Cubix and the 5,1 to co-operated.

 

[startergo]

???

Fingers firmly crossed. From the little info I've been able to gather online, it sounds like most of the issues people have had are related to using Nvidia GPUs with the Cubix and 5,1s - and all of the fresh hell, that driver limitations and incompatible OS versions have caused with that.

I'm hoping that the Radeon VIIs being supported natively in Mojave 10.14.6 (the single internally-mounted one I'm running at the moment, is running beautifully), will allow the Cubix and the 5,1 to co-operated.

I think it is mostly related to what switches are used inside the Cubix Xpander. I am using this board and have no issues with any PCIE card either video or NVME drives as long as they work inside the MacPro slots. My switch is:
PEX 8680 80-Lane, 20-Port PCIe Gen 2 Switch

 

[AlexMaximus]

Layout #1 is the best, I would go for that one. Most likely it's the layout you can expect the least amount of problems.
Contrary to some other folks here, I think you picked the best option for your scenario. The cost involved for the MP 7.1 is very hard to justify if your name is NOT Disney or Pixar. The cMP 5.1 has a lot of time left. With your setup, you have many options down the road. Mojave gives you a stable solution and pleasure for the next two years at least. After that, drivers in Catalina will be much more improved. Since the Dosdude firmware upgrade works great already, all you need to wait for is driver support for the 5700XT card (to even upgrade one more time) or better drivers for the current Vega VII to keep those cards. You actually have a real good solution for at least 3 - 4 years. In 2024 you can go and look for a used base 7.1 for a fraction of the current price AND keep the Cubix Expander AND keep all the internals as well. By then, the 28 core CPU will be super cheap and you can do the CPU swap yourself. Your business book looks so much better now with less dept, less invest and less pressure on you. If there is a crisis, you will be more stable. On the other hand, those highly theoretical benchmark numbers don't mean sh&%§. Rather invest excess money towards an excellent and sustainable relationship with your top clients. In the end, it's all about your offered product and service that you provide to your customers. It's NOT about the tool you use.

 

[bsbeamer]

Putting this out there... all available processors for MP5,1 are EOL'd by Intel and no longer getting microcode updates for mitigation of security vulnerabilities. The hardware itself may continue to work and there may be workarounds to get to Catalina or higher, but this security vulnerability is an issue that many are not aware of. It will only get worse over time. Don't know any client who would agree to continue using these machines for their projects if they were aware of this issue.

 

[Grumply]

Layout #1 is the best, I would go for that one. Most likely it's the layout you can expect the least amount of problems.
Contrary to some other folks here, I think you picked the best option for your scenario. The cost involved for the MP 7.1 is very hard to justify if your name is NOT Disney or Pixar. The cMP 5.1 has a lot of time left. With your setup, you have many options down the road. Mojave gives you a stable solution and pleasure for the next two years at least. After that, drivers in Catalina will be much more improved. Since the Dosdude firmware upgrade works great already, all you need to wait for is driver support for the 5700XT card (to even upgrade one more time) or better drivers for the current Vega VII to keep those cards. You actually have a real good solution for at least 3 - 4 years. In 2024 you can go and look for a used base 7.1 for a fraction of the current price AND keep the Cubix Expander AND keep all the internals as well. By then, the 28 core CPU will be super cheap and you can do the CPU swap yourself. Your business book looks so much better now with less dept, less invest and less pressure on you. If there is a crisis, you will be more stable. On the other hand, those highly theoretical benchmark numbers don't mean sh&%§. Rather invest excess money towards an excellent and sustainable relationship with your top clients. In the end, it's all about your offered product and service that you provide to your customers. It's NOT about the tool you use.

Yeah, I was pretty mad about the whole situation, having found it harder and harder to keep up with modern raw video files on my previous 5,1 (12-core 3.46Ghz, 64GB RAM, 12Gb Titan X). Especially with the insane amount of time I sunk into plotting out the 7,1 purchase (and the PC alternatives).

But thanks to the geniuses on these forums making it possible to both downvolt the Radeon VIIs, and enabling hardware acceleration for h264/h265 encoding/decoding for them as well - that has genuinely changed the game a bit.

With the single Radeon VII I'm running internally at the moment, and even with just 4x 6TB 3.5" SATA drives in a RAID 10 I'm still getting real-time playback of dual-stream 16-bit 4k Sonyraw footage and one stream of 6k 2.1Gbps Canon Raw in Davinci (even with a light application of temporal noise reduction applied). Which is kind of amazing.

I can also (thanks to the hardware video decoding mods) now playback and edit 4k h265 400mbps footage from my Fuji XT3s seamlessly in real-time - where my previous 5,1 build was completely unusable.

I can actually playback hardcore 4k 120fps HEVC encoded files in realtime now, and 8k youtube content is gorgeously smooth too (which was most certainly NOT the case with my previous 5,1, which was a beefy machine in its own right).

Once I've added in the 2nd Radeon VII and the enhanced storage speed via the 6-drive SATA RAID and the NVME cache drive, I expect things to get even easier - which should indeed help to hold me over for at least another two (maybe three) years, if need be.

And because there's so little actual sunk cost into the 5,1 itself (almost all of the components I'll be able to port directly over to a new machine eventually - and get much better performance from them when I do) I've been able to make my peace with it.

 

[DEH]

I think it is mostly related to what switches are used inside the Cubix Xpander. I am using this board and have no issues with any PCIE card either video or NVME drives as long as they work inside the MacPro slots. My switch is:
PEX 8680 80-Lane, 20-Port PCIe Gen 2 Switch

I am curious what Host Bus-to-Expansion Cable Adapter did you use? Have a picture of the setup? Box used?

Thanks

 

[zedex]

HA HA ..and to think I missed the chance to weigh in on the answer to this question at the time!! (HUGE Cubix fan here )

Drum roll please..... the CORRECT answer to your neatly bounded question is 3 (i.e. the third config outlined in your XL spreadsheet)

As there are only 4 configurations contemplated in your multiple choice quiz - we can mathematically determine that configs 1,2 and 4 are INCORRECT (maximum possible lane usage of approximately 26 lanes from an available 36).

The third config brings maximum possible lane usage to approximately 32 lanes from an available 36.

In reality, there is a 5th optimal config that saturates all 36 lanes....... but I haven't worked it out yet!

 

[startergo]

I am curious what Host Bus-to-Expansion Cable Adapter did you use? Have a picture of the setup? Box used?

Thanks

I will make a picture when I get back home. I know it is Nvidia cable.

 

[zedex]

I only learned of the switch type (brand/model etc) inside my CUBIX courtesy of the genuinely impressive tool developed by @joevt

It maps it's way through ALLLL your cool PCIe tech (txt output file attached).

It's like the System Info_PCI Panel on STEROIDS

 

[joevt]

I only learned of the switch type (brand/model etc) inside my CUBIX courtesy of the genuinely impressive tool developed by @joevt

It maps it's way through ALLLL your cool PCIe tech (txt output file attached).


It's like the System Info_PCI Panel on STEROIDS

It's just pciutils mostly with some extra formatting.

Looks like your pci ids database is out of date. Try sudo update-pciids

 

[startergo]

I am curious what Host Bus-to-Expansion Cable Adapter did you use? Have a picture of the setup? Box used?

Thanks

 

[joevt]

View attachment 1827815 View attachment 1827816 View attachment 1827814

Is that an x8 cable? Actually, it looks like it could be wider than an x8 cable.

The Netstor NA255A comes with two PCIe 3.0 x8 cables (1.5m each) to make an x16 connection. Of course, my MacPro3,1 outputs PCIe 2.0 x16 but that happens to be a similar bandwidth to the PCIe 3.0 x8 slots. I can disconnect one of the cables and it will work as an x8 connection.

 

[startergo]

Is that an x8 cable? Actually, it looks like it could be wider than an x8 cable.

The Netstor NA255A comes with two PCIe 3.0 x8 cables (1.5m each) to make an x16 connection. Of course, my MacPro3,1 outputs PCIe 2.0 x16 but that happens to be a similar bandwidth to the PCIe 3.0 x8 slots. I can disconnect one of the cables and it will work as an x8 connection.

Seller's description:

Add 5 PCI-E Gen 2 x16 slots to your system!

This is a professional grade expansion system that can fix into standard ATX form factor chassis! It is a more flexible choice compare to Cubix Xpander or NetStor PCIE expansion solutions.

I started decommissioning some of these PCI-E expansion boards. They are Gen 2 x16 expansion adapters and backplanes that can support up to 80 Gbps interconnection speed. (about 10 GB/s)

Solid and professional solution. You can mount it to standard ATX chassis. Note that the default PSUs used in these systems are Silverstone 1500W so you need to make sure your PSU has similar specifications.

Excellent for virtualization / graphics and video editing / crypto mining farm. Support 4x double with graphics card.

NOTE: You must populate all ATX 24-pin + 6-pin PCI-E + 8-pin M/B + 4-pin M/B power connectors on the backplane!

NOTE: Cyclone states this product is Operating System independent so should be compatible with Mac Pro. Please confirm with cyclone microsystem.

NOTE: while 0.3m cable provides better signal strength, you can get your own cable up to 3m.

Manual: available on cyclone website

Features
Gen2 x16 Upstream Port / Express Cable Interface from Host
Up to Three Meter Cable from Host
Five Gen2 PCIe Expansion Slots
two x16 PCI Express Slot (Double Width)
one x16 PCI Express Slot
two x16 PCI Express Slot ( x8 electrical)
One 80 Lane PCI Express Switches supporting:
Peer-to-Peer Communications
Non-Blocking Switch Fabric
Data Integrity
Quality of Service
ATX Form Factor

Package
Includes:
- 1x PCIe2-426 host adapter (install to your computer)
- 1x PCIe2-427 backplane
- 1x Molex 111008-0114 (~0.3 meter) PCI-E iPass x16 cable

Cyclone Microsystems 5-Slot PCI Express Gen 2 x16 Expansion System Kit Mac PC • $899.00

CYCLONE MICROSYSTEMS 5-SLOT PCI Express Gen 2 x16 Expansion System Kit Mac PC - $899.00. FOR SALE! 142860125528

picclick.com

It is a 1x Molex 111008-0114 (~0.3 meter) PCI-E iPass x16 cable
Nvidia p/n 030-0237-000

Nvidia 030-0237-000 Tesla GPU HIC Host Interface Cable molex 111008-0114 .5M

Nvidia 030-0237-000 Tesla GPU HIC Host Interface Cable molex 111008-0114 .5M

www.goltukh.top