Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
2.6GB/s (2600MB/s) RAID array transfer rate via Thunderbolt 2
- Thread starter FireWire2
- Start date
- Sort by reaction score
if possible which hard drives manufacture and model
raid per device
raid software
same or not same TB2 controller
can this be tested 3hrs+ to several days with same result if not why feel do not need more speed
may be better to add a third device to the mix to increase read write if the several hour/day test fails
in my setup i have to benchmark for week+ as the units will never sleep as this is video editing you may be ok with 1 day testing
Fastest Thunderbolt RAID!
Mission accomplished! Fastest Thunderbolt RAID (for now, lol)
Since i do not know NLE process much, therefore I wont go there...
But my customer - Burbank in CA, near Hollywood, he wants 2600MB/s (big B) and as much as storage that I can get for him.
After weeks of struggle to come up with solution, last Friday we're able to get 2600MB/s consistently with RAID50 - (not RAID60) to new a MAC Pro using 2x Thunderbolt ports connect to two 12x drives thunderbolt raid boxes from DATOptic
We tried Areca, Promise Tech thunderbolt 2 enclosures with the same setup, same HDD, where ARECA has a little bit of fan noise, but not Pegasus2 R8
With 3x boxes of AREAC or Pegasus2 R8 as RAID50, we're able to get the goal transfer rate. But our goal is only use TWO TB2 ports
Here come the rescue.
I called DATOptic and told them the issue. They offer me 100% money back guarantee in 30 days, but I have to do all the tests, which is fine by me.
As you can see the previous posted, the only combo that get us 1300MB/s plus is T12-S6.TB2 (including 2m TB cable) with 12x Seagate 4TB 5900 rpm HDD
I guess 1TB plate does make the different.
After debate and read through this post we decide to try RAID50/60.
It appears RAID50 lot more evenly in both Read and Write. This make sense because RAID60 requires lots more I/O Processor resource
The RAID5 is handle by the T12-S6.TB2
We create 2x RAID5 form DATOptic boxes via WebGUI.
The Web GUI exactly the same as Areca, turn out DATOptic use Areca RAID engine.
With two volumes, we stripe these arrays in DU to create a RAID50
One mistake we made at first, that is using Thunderbolt port 1 and 3.
We can not get over 1400MB/s, turned out we connected to the
same TB2 bus.
Here is ref for Tbolt2 bus in nMP
http://www.tekrevue.com/tip/mac-pro-thunderbolt-performance/
And testing the hot plug of the unit, we simply move the cable from port #3 port to port #2 port. It takes about 30~45sec the volume appears back to the system
No reboot required, even with 24x HDD and 8x fans are running, there is hardly fan noise, very quiet.
Here are the test results of RAID50 with 88TB usable capacity:
AJA
Quick Bench:
Avid Test
Here are the test of RAID60 with 80TB usable capacity:
Quick Bench:
Avid Test
Mission accomplished! Fastest Thunderbolt RAID (for now, lol)
Since i do not know NLE process much, therefore I wont go there...
But my customer - Burbank in CA, near Hollywood, he wants 2600MB/s (big B) and as much as storage that I can get for him.
After weeks of struggle to come up with solution, last Friday we're able to get 2600MB/s consistently with RAID50 - (not RAID60) to new a MAC Pro using 2x Thunderbolt ports connect to two 12x drives thunderbolt raid boxes from DATOptic
We tried Areca, Promise Tech thunderbolt 2 enclosures with the same setup, same HDD, where ARECA has a little bit of fan noise, but not Pegasus2 R8
With 3x boxes of AREAC or Pegasus2 R8 as RAID50, we're able to get the goal transfer rate. But our goal is only use TWO TB2 ports
Here come the rescue.
I called DATOptic and told them the issue. They offer me 100% money back guarantee in 30 days, but I have to do all the tests, which is fine by me.
As you can see the previous posted, the only combo that get us 1300MB/s plus is T12-S6.TB2 (including 2m TB cable) with 12x Seagate 4TB 5900 rpm HDD
I guess 1TB plate does make the different.
After debate and read through this post we decide to try RAID50/60.
It appears RAID50 lot more evenly in both Read and Write. This make sense because RAID60 requires lots more I/O Processor resource
The RAID5 is handle by the T12-S6.TB2
We create 2x RAID5 form DATOptic boxes via WebGUI.
The Web GUI exactly the same as Areca, turn out DATOptic use Areca RAID engine.
With two volumes, we stripe these arrays in DU to create a RAID50
One mistake we made at first, that is using Thunderbolt port 1 and 3.
We can not get over 1400MB/s, turned out we connected to the
same TB2 bus.
Here is ref for Tbolt2 bus in nMP
http://www.tekrevue.com/tip/mac-pro-thunderbolt-performance/
And testing the hot plug of the unit, we simply move the cable from port #3 port to port #2 port. It takes about 30~45sec the volume appears back to the system
No reboot required, even with 24x HDD and 8x fans are running, there is hardly fan noise, very quiet.
Here are the test results of RAID50 with 88TB usable capacity:
AJA
Quick Bench:
Avid Test
Here are the test of RAID60 with 80TB usable capacity:
Quick Bench:
Avid Test
Last edited:
Great job! If size and noise is a concern, maybe it was possible reaching these speeds with about six SSD drives.
Couldn't you use two of the Sonnet thunderbolt-to-PCIe expansion chassis' and insert two OCZ Technology ZD4CM88 PCIe 3.2TB SSD's in a RAID 0 setup? Each drive has nearly a 2.8GB/s speed for both seq read & write. The combined theoretical max performance would get you nearly 5.4GB/s read/write speeds wouldn't it? Or is there some flaw in this idea that I am unaware of?
It would cost around $40k though
I have absolutely no experience or knowledge in this arena, however. Just wondering if it would even be possible to do that.
It would cost around $40k though
I have absolutely no experience or knowledge in this arena, however. Just wondering if it would even be possible to do that.
Last edited:
Nice to see you got it sorted, it's quite an achievement, although I still have my reservations as to what workflow would require so much speed!
Best part I think is that you can pull one of the cables and have it return to normal so easily. I thought it would spell chaos for a RAID so I really didn't expect this.
I would just make sure you have those TB cables anchored well - they are prone to falling out with minimal effort as obviously there's no clip mechanism.
Especially when the nMP is designed to be lifted up and turned around when plugging other cables in.
Does this setup still enable your client to have the required number of monitors?
And also what model HDDs did you go with?
----------
^^^ To the few posters above, that would only have left the client a few minutes of storage for their footage - the setup he went with has 88TB capacity.
Best part I think is that you can pull one of the cables and have it return to normal so easily. I thought it would spell chaos for a RAID so I really didn't expect this.
I would just make sure you have those TB cables anchored well - they are prone to falling out with minimal effort as obviously there's no clip mechanism.
Especially when the nMP is designed to be lifted up and turned around when plugging other cables in.
Does this setup still enable your client to have the required number of monitors?
And also what model HDDs did you go with?
----------
^^^ To the few posters above, that would only have left the client a few minutes of storage for their footage - the setup he went with has 88TB capacity.
Can not help at nMP end but with T12-S6.TB2 the TB cable is pretty well off
It has a bracket to hold and support the TB cable - I will get a image of that if you want
Yes, we only use two TBolt2 bus (2x ports) there are 4x ports are available to use, but mainly #5 and #6
Yes 88TB
We did get couples extra HDD (ST4000DM000) for spare
In RAID GUI we set Rebuild with Blank HDD, so there is no need to set the HDD as hot spare b4 the raid can rebuild.
we did conduct a failed HDD test by remove a HDD (hot unplug)
Then insert a new HDD, within 45 seconds the RAID start rebuild.
The only thing we do not know how good/reliable the HDD drive ST4000DM000 is, but base on review from newegg and amazon. We feel comfortable to use it
----------
Couldn't you use two of the Sonnet thunderbolt-to-PCIe expansion chassis' and insert two OCZ Technology ZD4CM88 PCIe 3.2TB SSD's in a RAID 0 setup? Each drive has nearly a 2.8GB/s speed for both seq read & write. The combined theoretical max performance would get you nearly 5.4GB/s read/write speeds wouldn't it? Or is there some flaw in this idea that I am unaware of?
It would cost around $40k though
I have absolutely no experience or knowledge in this arena, however. Just wondering if it would even be possible to do that.
Yes you can do that but there will be very little storage space and potential data lost due to RAID0, no redundant
Last edited:
If you replace the 4TB HDDS with He-filled 6 TB HDDS, you may give him more storage and a higher speed.
Couldn't you use two of the Sonnet thunderbolt-to-PCIe expansion chassis' and insert two OCZ Technology ZD4CM88 PCIe 3.2TB SSD's in a RAID 0 setup? Each drive has nearly a 2.8GB/s speed for both seq read & write. The combined theoretical max performance would get you nearly 5.4GB/s read/write speeds wouldn't it? Or is there some flaw in this idea that I am unaware of?
It would cost around $40k though
I have absolutely no experience or knowledge in this arena, however. Just wondering if it would even be possible to do that.
I'm really surprised that you went with 12-drive RAID-5 slices. Virtually any "best practices" white paper will strongly recommend no larger than 6-drive slices (5 data plus one parity).
Not only for performance, but for the simple fact that a 4TB drive is dangerously close to the expected error rate for the drive. The desktop ST4000DM000 is rated at an expected error rate of one unrecovered read error per 10^14 bits. That basically means that Seagate is telling you that you cannot rebuild your RAID5 slice - you can expect an error from at least one of the 11 drives before you can rebuild.
And why on earth would you choose consumer level ST4000DM000 drives? I keep my porn libraries on Constellation ES.3 drives, because the consumer drives are incompatible with RAID controllers.
Seriously, ST4000DM000 drives for a money generating setup? Seriously!?
I have a few hundred TB of RAID storage, and most of it is using 6-drive RAID-60 slices (4 data plus two parity), and they're on expensive enterprise class Savio drives. The rest are scratch volumes at RAID-0 for performance - but still on enterprise Savios.
At the risk of repeating myself - you're seriously going to recommend consumer grade non-RAID-qualified ST4000DM000 drives to your "deep pocket" client?
Abort! Redo it with Constellation ES.3 drives. Consider using 5 drive RAIDs and having two hot spares.
Edit: Constellation ES.4 drives are out now - and their expected error rate is 10 times lower than the ST4000DM000 drives.
Last edited:
Congrats on the getting the numbers he was after. I don't have the final tally but seems as if he is 10K in! To spend 10K (not including your fee) to edit 4K video is, let's say ! Next week the moderators at Avid get the up coming MC8 that should have the new engine.
When MC8 with the new engine is available to the "general public", I will post here results pertaining to 4K editing!
I do understand you were getting paid for the "mission" and any detour might have cut you out of the loop.
10K + to edit 4K=
! Next week the moderators at Avid get the up coming MC8 that should have the new engine. When MC8 with the new engine is available to the "general public", I will post here results pertaining to 4K editing!
I do understand you were getting paid for the "mission" and any detour might have cut you out of the loop.
10K + to edit 4K=
Whoa, those Seagate drives are a worry.
Yes technically you can use them and they will work - I've done it before as well - but with the knowledge that they would be thrown out of the RAID, and it did of course eventually happen - even in a much smaller 8 drive array.
Different manufacturers use different acronyms - TLER, ERC, CCTL - but have a search around for info on why you should use drives with error recovery designed for RAID systems.
Yes technically you can use them and they will work - I've done it before as well - but with the knowledge that they would be thrown out of the RAID, and it did of course eventually happen - even in a much smaller 8 drive array.
Different manufacturers use different acronyms - TLER, ERC, CCTL - but have a search around for info on why you should use drives with error recovery designed for RAID systems.
Firstly wow, that's a seriously impressive, it's actually made me rethink my decision to stick with my 5,1!!
I do however concur with all the disk talk it's a scarey amount of data and disks when it only needs two or three to fail . Please make sure it gets backed up and is on a decent UPS. I think I would of stuck with smaller, more proven enterprise disks and then moved over to 4tb when they have been around a little longer and the plants are more established. Coming from someone who recently lost a pair or 2tb disks from a purchased batch of seven within a week of each other.
Nox
I do however concur with all the disk talk
it's a scarey amount of data and disks when it only needs two or three to fail . Please make sure it gets backed up and is on a decent UPS. I think I would of stuck with smaller, more proven enterprise disks and then moved over to 4tb when they have been around a little longer and the plants are more established. Coming from someone who recently lost a pair or 2tb disks from a purchased batch of seven within a week of each other.Nox
Yeah, as I mentioned, I run tech at a post house, I work with these formats all the time, and I cannot figure out what sane workflow this requirement is intended to serve. The only thing that makes any sense at all is that the client wants to transcode the R3D to a 16-bit uncompressed RGB format (would indeed be about 2600 MB/s) and work from that. But this is nuts, and there are a zillion ways to avoid it — transcode to 'only' 4K (the highest resolution you can deliver at anyway), transcode to something like ProRes 4444 or the new XQ (they're designed for stuff like this; I watch ProRes content on 30+ foot screens all the time and it holds up fine), work directly from the raw files, etc.
The client seems to have settled on a completely impractical workflow (footage will run about 10 terabytes per hour) and is having OP jump through crazy hoops to accommodate it, rather than stepping back and asking if there might be better approaches.
One thing that I know and practice... That most of IT people omitted
Check and mapped bad sector(s) from all HDD
I'd doing that for years on my RAID5 with Seagate 3TB HDD, it is in a fifth yrs of service.
In other hand, we also have back up already implement.
In 17 yrs of IT consultant, I saw many BSOD evens with enterprise/desktop HDD. Both enterprise and desktop has their fair share of it.
The thing is if you do not set and maintenance the array correctly. It will failed. Which most pp do, that is where I'm coming to the picture
So far over 100's RAID set up that I done, couple of them lost data, due to remove wrong drive(s). None of them lost data due to BAD drive
Check and mapped bad sector(s) from all HDD
I'd doing that for years on my RAID5 with Seagate 3TB HDD, it is in a fifth yrs of service.
In other hand, we also have back up already implement.
In 17 yrs of IT consultant, I saw many BSOD evens with enterprise/desktop HDD. Both enterprise and desktop has their fair share of it.
The thing is if you do not set and maintenance the array correctly. It will failed. Which most pp do, that is where I'm coming to the picture
So far over 100's RAID set up that I done, couple of them lost data, due to remove wrong drive(s). None of them lost data due to BAD drive
Last edited:
Since you built his system I am sure you will be closely watching and maintaining it. If possible push the system to the limits in Avid by 4K multicam. There is a 9 cam max in MC. Post results pertaining to full and half rez. State what was the max amount of cams and at what rez.
Also, if the max is reached within 4-6 cams, test again when the new engine is in MC8.
Of course it is not necessary to wait for a 4K multicam shoot. Attached is a screen shot using "5 cam" in HD. Simply edit the footage into x-amount of parts to simulate x-amount of cameras.
Also, if the max is reached within 4-6 cams, test again when the new engine is in MC8.
Of course it is not necessary to wait for a 4K multicam shoot. Attached is a screen shot using "5 cam" in HD. Simply edit the footage into x-amount of parts to simulate x-amount of cameras.
Attachments
Last edited:
Congrats but 1~2 drive over 24 hdd is risky i strongly thing you need to do a 60 or 3 devices with a 60 to get the speeds you need and also add another 30TB storage
still fiber would have been so much simpler then this but nice job
* i said 1~2 as he can lose only 1 hard drive per device and be able to rebuild but can not louse 2 dives on same device as currently setup vary risky with number of drives here
still fiber would have been so much simpler then this but nice job
* i said 1~2 as he can lose only 1 hard drive per device and be able to rebuild but can not louse 2 dives on same device as currently setup vary risky with number of drives here
My RAID controllers do that automatically - scanning for marginal sectors whenever the drive is idle and remapping them if any are found.
This must be a Time Machine drive, since Seagate didn't release 3TB disks until spring 2011
Yes, lot of user does not use this function
Oppps, It was the 2TB, we change to 3TB later in this NAS, since 2010
http://www.tomshardware.com/forum/265641-32-40tb-server-performance-issue
Wow, impressive !
But, DATOptic, really ?
My experience is very limited, as I bought from them only once - one time too many - but my guess would be lowest bidder components and zero support after the initial effort to sell you those things .
Parts so cheap and obscure it'll take a genius to keep them compatible, let alone running at a good pace .
But, DATOptic, really ?
My experience is very limited, as I bought from them only once - one time too many - but my guess would be lowest bidder components and zero support after the initial effort to sell you those things .
Parts so cheap and obscure it'll take a genius to keep them compatible, let alone running at a good pace .
I think a recent report indicates that seagate have some of the highest failure rates in a 24/7 environment. http://blog.backblaze.com/2014/01/21/what-hard-drive-should-i-buy/
A couple of interesting quotes from that link:
- Basically, we buy the least expensive drives that will work.
- We are focusing on 4TB drives for new pods. For these, our current favorite is the Seagate Desktop HDD.15 (ST4000DM000).
- (in a comment) It looks like exactly one model/size of Seagate had a high failure rate, and the rest did not. That's actually believable, because every hard drive company has bad models from time to time.
Last edited:
Yes at work we stopped buying seagate altogether (max of 5 drives a year) as we buy 100 to 300 drives a year and seagtes failed to fast for them to be usable regardless of what series and as that link shows WD is the best over all
Hitachi is also a good drive too but they are limited to general uses primarily
while he does have the speed i do think the risk of data loss on this project is to high and as he used seagates i do think enough will fail for him to understand why i don't buy them except to see if they have any thing better or as a last resort
also do not buy HDD cheap you will pay for it in long run
Given that the new Seagates were only in service for 3 to 4 months at the time of writing I wouldn't put too much stock on the failure numbers as yet. Also the 36 month view of drives only 74% of Seagate drives were operational compared to 97% of Hitachi that is a bit of a concern.
In a date center environment with raid and hot spares and the number of drives they buy it makes sense to get the cheapest but professional use I would be much more careful.
the trend of seagates has been true for over a decade sad but true i do not think it will change
the reason why WD and Hitachi do not fail is they use perpendicular writing a technology that wd and Hitachi mainly H hold all patents to and wd leases this is why they last so much longer due to this there is not much hope in seagates ever getting better
there is 2 versions of data centers have the most data, have the data most reliable if you are smart you do both
ie if you have 4 NAS divices you should be able to lose 1 whole nas and have ZERO data loss and 1 hdd per 6 should be able to lost at all times
depending on what data center you work at they will have a version of the above due to the kind of data we hold this is what we make as our specs yes it means that we have 25%+ of redundant data at all times