Sidebar Sidebar
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.

2.6GB/s (2600MB/s) RAID array transfer rate via Thunderbolt 2

FireWire2

FireWire2

macrumors 6502
Original poster
Oct 12, 2008
363
6
One of mine client, who wants to editing 4K video with AVID.
- less than 15ms latency
- support 4K @ 50fps
- Red camera raw footage
Base on his calculation, he needs RAID5/6 thunderbolt solution that can transfer around 2600MB+ per second to his new MAC Pro for 4K editing.
Note: There are 2x TB monitors connect to this nMP too

Any Idea how to get this done?

TIA

UPDATE:
Mission accomplished! Fastest Thunderbolt RAID (for now, lol)

Since i do not know NLE process much, therefore I wont go there...:)

But my customer - Burbank in CA, near Hollywood, he wants 2600MB/s (big B) and as much as storage that I can get for him.
After weeks of struggle to come up with solution, last Friday we're able to get 2600MB/s consistently with RAID50 - (not RAID60) to new a MAC Pro using 2x Thunderbolt ports connect to two 12x drives thunderbolt raid boxes from DATOptic

We tried Areca, Promise Tech thunderbolt 2 enclosures with the same setup, same HDD, where ARECA has a little bit of fan noise, but not Pegasus2 R8

With 3x boxes of AREAC or Pegasus2 R8 as RAID50, we're able to get the goal transfer rate. But our goal is only use TWO TB2 ports

Here come the rescue.
I called DATOptic and told them the issue. They offer me 100% money back guarantee in 30 days, but I have to do all the tests, which is fine by me.

As you can see the previous posted, the only combo that get us 1300MB/s plus is T12-S6.TB2 (including 2m TB cable) with 12x Seagate 4TB 5900 rpm HDD

I guess 1TB plate does make the different.

After debate and read through this post we decide to try RAID50/60.

It appears RAID50 lot more evenly in both Read and Write. This make sense because RAID60 requires lots more I/O Processor resource

The RAID5 is handle by the T12-S6.TB2
We create 2x RAID5 form DATOptic boxes via WebGUI.

The Web GUI exactly the same as Areca, turn out DATOptic use Areca RAID engine.

With two volumes, we stripe these arrays in DU to create a RAID50
ry%3D400


One mistake we made at first, that is using Thunderbolt port 1 and 3.
We can not get over 1400MB/s, turned out we connected to the
same TB2 bus.

Here is ref for Tbolt2 bus in nMP
http://www.tekrevue.com/tip/mac-pro-thunderbolt-performance/
And testing the hot plug of the unit, we simply move the cable from port #3 port to port #2 port. It takes about 30~45sec the volume appears back to the system

No reboot required, even with 24x HDD and 8x fans are running, there is hardly fan noise, very quiet.

Here are the test results of RAID50 with 88TB usable capacity:

AJA

ry%3D480


Quick Bench:

ry%3D400


Avid Test

ry%3D480

Here are the test of RAID60 with 80TB usable capacity:

Quick Bench:

ry%3D400


Avid Test

ry%3D480
 
Last edited:
FireWire2 said:
One of mine clients, who wants to editing 4K video with AVID.
- less than 15ms latency
- support 4K @ 50fps
- Red camera raw footage
Base on his calculation, he needs RAID5/6 thunderbolt solution that can transfer around 2600MB+ per second to his new MAC Pro for 4K editing.
Note: There are 2x TB monitors connect to this nMP too

Any Idea how to get this done?

TIA
Click to expand...

Apparently you can daisy chain Lacie's drives to achieve 3000 MB/s transfer rates. https://www.lacie.com/us/products/product.htm?id=10623

Scroll down to the info box where it shows them daisy chained.

Looks like you might also be able to do the same with their SSD Raid system. https://www.lacie.com/us/products/product.htm?id=10621
 
Your client is hallucinating.

High end enterprise systems on 24 Gbps SAS can hit about 2.1 * 10^9 bytes per second. RAID-0 striping across at least two 24 Gbps SAS links (and dozens of spinning drives on each link) can hit your client's perceived demand. Oops - no SAS on the new Mac Pro. Fail.

A RAID-0 across a pair of top-end PCIe SSDs could hit it too - but the new Mac Pro doesn't have PCIe slots. Fail again.

First step - have your client work with Red and Avid to come up with realistic requirements.

Second step - find a dual socket workstation with 80 PCIe lanes for your client.

Raddock said:
Apparently you can daisy chain Lacie's drives to achieve 3000 MB/s transfer rates. https://www.lacie.com/us/products/product.htm?id=10623

Scroll down to the info box where it shows them daisy chained.
Click to expand...

That 3GB/sec is aggregate to multiple RAID arrays, not the speed to a single array. If you could do software RAID-0 in the host across the arrays you could get the aggregate to a single filesystem, but at the expense of reliability.
 
Last edited by a moderator:
FireWire2 said:
One of mine clients, who wants to editing 4K video with AVID.
- less than 15ms latency
- support 4K @ 50fps
- Red camera raw footage
Base on his calculation, he needs RAID5/6 thunderbolt solution that can transfer around 2600MB+ per second to his new MAC Pro for 4K editing.
Note: There are 2x TB monitors connect to this nMP too

Any Idea how to get this done?

TIA
Click to expand...

50fps at 4k in .r3d, even at 6:1 is only 97 MB/s.
 
Honestly, I'm scratching my head as well.

Why would he ever need 2.6GB/s transfer rates?

REDCODE offers the following compression rates (5K RED Epic).
18:1 = 1.48 GB/Minute
15:1 = 1.77 GB/Minute
10:1 = 2.65 GB/Minute
8:1 = 3.34 GB/Minute (Default setting)
5:1 = 5.26 GB/Minute
3:1 = 8.70 GB/Minute

Even at 3:1 (23.98fps) you only need around 150MB/s sustained per clip.
 
Thank you all for the inputs. Appreciated very much

Pressure
I'm an IT guy.. don't know what he wants to do.. but as far as clients concern, he is paying me to make it...I will try my best to earn that $$$ :)

Raddock
Lacie solution as you suggested, does not work, Thunderbolt 2 transfer rate is cap @ 1400MB/s

AidenShaw
Thank you for your suggestion, I will try a RAID60 and RAID50. As of now I can get 1300MB/s to 1400MB on RAID6/5 respectively
 
1400MB Read/Write Thunderbolt

Update:

I'm able to get:
1300MB/s RAID6 and 1400MB/s on RAID5 with new MAC Pro

It's not where my client wants
 
beaker7 said:
Did your client ask you to build a machine with storage speeds 25 times faster than they need?
Click to expand...

Yes, he wants an array @ 2600MB/s to his new MACPro. He has money to burn :)
We all know about "WANT and NEED", right?
 
FireWire2 said:
Yes, he wants an array @ 2600MB/s to his new MACPro. He has money to burn :)
We all know about "WANT and NEED", right?
Click to expand...

Sure, but you said that's what he calculated he NEEDS in order to edit that footage, which is demonstrably incorrect.
 
RED user and camera shop here. We have multiple RED Dragon (6K) cameras and I am very familiar with this problem. Your client is correct to want 2600 MB/s RAID performance. Editing 4K and working in RAW (6K) requires monstrous bandwidth. 1000 MB/s will not provide real-time continuous playback and edit performance. TB2 and the nMP is hopeless overall. TB3 will finally get us to what we need. Tell your client to either down step to a full-size MP with SAS to RAID 50 (recommend JMR Silverstor) and then wait for TB3 in x-years.
 
The transfer limit of Thunderbolt 2 is 20 Gbps/8 = 2500 MB/s. Thus, a single thunderbolt 2 connection can't achieve 2600 MB/s.

One suggestion: Get two Promise Pegasus2 R4's. Put four Crucial M550 (or another brand) 1T SSDs into each Promise Pegasus2 R4 (total 8 SSDs). Do not daisy-chain the two Promise Pegasus2 R4's. Connect them separately to two different TB ports on two different controllers on the nMP. Use the JBOD mode on each Promise Pegasus2 R4. If you can make a RAID0 out of these 8 SSDs, you can then achieve 2500 MB/s or above.

You may also consider using two LaCie LBD Thunderbot2's. Do not daisy-chain them. Connect them separately to two different TB ports on two different controllers on the nMP.

I plan to make a RAID0 or a RAID5 using four 512GB SSDs and a Promise Pegasus2 R4. I have got 4 SSDs (Toshiba Q Series Pro 512GB). But I am still waiting for the diskless Promise Pegasus2 R4 from the Apple Online Store.
 
Last edited:
blackwoodfx said:
RED user and camera shop here. We have multiple RED Dragon (6K) cameras and I am very familiar with this problem. Your client is correct to want 2600 MB/s RAID performance. Editing 4K and working in RAW (6K) requires monstrous bandwidth. 1000 MB/s will not provide real-time continuous playback and edit performance. TB2 and the nMP is hopeless overall. TB3 will finally get us to what we need. Tell your client to either down step to a full-size MP with SAS to RAID 50 (recommend JMR Silverstor) and then wait for TB3 in x-years.
Click to expand...


What he says......

although, I've been intrigued by the various upgrade options these guys have put together for the classic mac pro..if correct, that gets you near 2g rates

http://www.maxupgrades.com/istore/index.cfm?fuseaction=product.display&product_ID=354&ParentCat=414
 
FireWire2 said:
Raddock
Lacie solution as you suggested, does not work, Thunderbolt 2 transfer rate is cap @ 1400MB/s
Click to expand...


It should work, otherwise Lacie wouldn't quote it on their website.
I also saw a video of multiple Lacie Little Big disk2 s software raided together and it was faster than 1400 mb/s.

----------
blackwoodfx said:
RED user and camera shop here. We have multiple RED Dragon (6K) cameras and I am very familiar with this problem. Your client is correct to want 2600 MB/s RAID performance. Editing 4K and working in RAW (6K) requires monstrous bandwidth. 1000 MB/s will not provide real-time continuous playback and edit performance. TB2 and the nMP is hopeless overall. TB3 will finally get us to what we need. Tell your client to either down step to a full-size MP with SAS to RAID 50 (recommend JMR Silverstor) and then wait for TB3 in x-years.
Click to expand...

Hey, quick question: How is it that we need such a HUGE bandwith for dragon footage? Is the compression that much different from old EPIC footage?
I was always wondering, if the data rate of red/epic footage is so low on paper as it has been posted in this thread, where lies the bottleneck in my station? Is it the hard drives or really just processing power?

I've got a nMP with 8 cores and dual d700 connected to a Pegasus1 R6 with 12TB and am not able to playback epic footage at 1/2 without caching for a few seconds in redcine. Would I need a red rocket in order to play it back smoothley? Or is the pegasus 1 too slow with it's 680 mb/s read?
 
FireWire2 said:
Yes, he wants an array @ 2600MB/s to his new MACPro. He has money to burn :)
Click to expand...

Are you sure your client knows the difference between 2600MB/s and 2600Mb/s ? That the capital letter makes a difference. Folks with more money then sense may not know the difference. If they don't know that, part of being a professional is to tell them.... not to drain their bank account on ignorance.
 
FireWire2 said:
One of mine clients, who wants to editing 4K video with AVID.
...
Base on his calculation, he needs RAID5/6 thunderbolt solution that can transfer around 2600MB+ per second to his new MAC Pro for 4K editing.
Note: There are 2x TB monitors connect to this nMP too
Click to expand...

Again this teeters between "want" and "need" but 2 Thunderbolt monitors why? If client wants to maximize available Thunderbolt bandwidth then dropping those will help. If "require" that the monitors are dropped on a very long cable away from the nMP perhaps. "needs" the extra USB/FW/Ethernet ports ... perhaps ...

But if 'sunk costs' in TB monitors already .... if they have money to burn that shouldn't be a problem.


Likewise everything bound up in a single RAID 6 imagine.... requirements or simplification?


As a storage bus the TB v2 controllers are pragmatically about 1500MB/s conduits ( ~ 3x PCI v2 *** ) . Still have to trunk down to x4 PCIe v2 implementations to get on/off the Thunderbolt network. The upside is that there are three of busses in Mac Pro 2013. 3 * 1500MB/s ( 4,500 MB/s) means it is possible. It just isn't going to be some fixed, turn-key solution.

What looking for a RAID 0 solution layered on top of 2 (or 3 ) RAID-6 subsystems. Layering RAID 0 on top of virtual disks that are extremely unlikely to fail is a relatively very small additional risk.

If "require" a "one wire/cable" solution then no; there isn't one.


There is more than a little amount of self-inflicted pain here in editing directly in RAW 4-6K files here. There is gobs of redundant data being dragged across the storage subsystem probably for not alot of real necessity.




*** While the Thunderbolt v2 data bandwidth is 20Gb/s the bandwidth of PCIe specific data is not the same thing. Thunderbolt data packets carry PCIe data but they are not the same thing or bandwidth. To any one node on a Thunderbolt network the transmission speed to the host PCIe network is limited by the PCIe switching limits present in all TB controllers. All the data on a TB v2 doesn't have to all be bound for just one node on the network.
 
AidenShaw said:
Oops - no SAS on the new Mac Pro. Fail.
Click to expand...

http://www.areca.us/products/thunderbolt2.htm


There is a Fail here ..... just not on the Mac Pro side. There is huge difference between there not being a being able to order directly from Apple's store inventory and there being no SAS solutions.

Will need more than one, but since this thread is an "unrestricted budget" exercise... that isn't an issue.
 
Fastest Thunderbolt RAID

Just an update:

We'd test ARECA, Promise tech Tech, and DATOptic thunderbolt RAID enclosure

So far the only ONE enclosure that can delivery consistently over 1300MB/s

We are going to create a RAID50/RAID60 out of 2x identical RAID volume

I think we can get more 2600MB/s
We are waiting for the 2nd unit coming this week

Can not wait
 
Last edited:
AidenShaw said:
Your client is hallucinating.

High end enterprise systems on 24 Gbps SAS can hit about 2.1 * 10^9 bytes per second. RAID-0 striping across at least two 24 Gbps SAS links (and dozens of spinning drives on each link) can hit your client's perceived demand. Oops - no SAS on the new Mac Pro. Fail.

A RAID-0 across a pair of top-end PCIe SSDs could hit it too - but the new Mac Pro doesn't have PCIe slots. Fail again.

First step - have your client work with Red and Avid to come up with realistic requirements.

Second step - find a dual socket workstation with 80 PCIe lanes for your client.



That 3GB/sec is aggregate to multiple RAID arrays, not the speed to a single array. If you could do software RAID-0 in the host across the arrays you could get the aggregate to a single filesystem, but at the expense of reliability.
Click to expand...

My thoughts exactly. Personally, I thought his client was doing drugs.
 
TheHerdForever said:
My thoughts exactly. Personally, I thought his client was doing drugs.
Click to expand...

That was my original though, but after he ran some latency test from Sony and AVID...
@ 1300MB/s it supported 2K only... I guess he does not use compress codec

blackwoodfx said:
RED user and camera shop here. We have multiple RED Dragon (6K) cameras and I am very familiar with this problem. Your client is correct to want 2600 MB/s RAID performance. Editing 4K and working in RAW (6K) requires monstrous bandwidth. 1000 MB/s will not provide real-time continuous playback and edit performance. TB2 and the nMP is hopeless overall. TB3 will finally get us to what we need. Tell your client to either down step to a full-size MP with SAS to RAID 50 (recommend JMR Silverstor) and then wait for TB3 in x-years.
Click to expand...

I guess he is right
 
Last edited:
You must log in or register to reply here.
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.
Top Bottom
Back