Upgrading RAID for video editing - which card?

[nanofrog]

First RAID3 test with AJA: (screenshot attached)
I chose read/write 16GB test with video frame 1920x1080 8-bit.

Not terrible, but I'd have hoped for more.

Of course, you can always take one of the Hot Spares, and make it an active disk (leaving 1x Hot Spare) via Online Expansion.

But I still think you'd be better off with RAID 5 (same fail-over as RAID 3).

BTW, you don't work with 10bit?
And test with the cache active, and watch what happens.

Here's Xbench 1.3:

This is all prior to swapping any cables around, of course.

Go ahead and see what happens performance-wise after swapping cables (move it to each port; shut downs between them of course...).

This is my current RAID and volume setup. I'm going to shut down and switch the internal cable to another port first, and see what changes.

Good luck, and don't forget to try each port (should tell you which internal group is sharing with the external ports, all things being equal - same configuration <array level, member count, and disks used>, same test).

 

[wonderspark]

I feel dumb, but I cannot get the internal cable to release from the card. It was tight going on, and I'm afraid the port is going to break off if I pull harder. How do you release an SFF-8087 connector?

 

[wonderspark]

I can get the cable that came with the card to go on and off pretty easy. This PC-PitStop cable is really locked in there. Gah!

 

[wonderspark]

Eff this. I guess it's a permanent part of the card now. I'll have to wait a few days for a second cable to show up in the mail.

 

[nanofrog]

I can get the cable that came with the card to go on and off pretty easy. This PC-PitStop cable is really locked in there. Gah!

There are variances to both connectors (card and cable), so they can be tight from time to time.

Just make sure the locking tab is sufficiently depressed, and be cautious (they are fairly stout, but can be broken). Rocking/wiggling side to side while tugging on it might help too (could tell you if the locking tab is sufficiently depressed, though usually you'll feel the connector slip backwards when it is due to a spring action between the tab and retaining clip).

 

[wonderspark]

I tugged, rocked, wiggled and everything for probably a half hour. It's in there good. I put it all back together because I heard some unsettling popping and snapping sounds as I tugged the crud out of it, and I wanted to make sure I didn't break something. It still works, but I've given up on it.

I did order another one, so when that comes in, I'll re-test the port switching issue.

For now, I pulled the UPS plug, and was alarmed to see the battery drop to 40% in about ten seconds. I think the big UPS (SUA1500) needs a new battery. I bought this used from a company that was going out of business for $25, and that was December 2009, so it's an old battery for sure. I pulled the smaller UPS (BackUPS RS 1500) that only has monitors and external drives hooked up, and it's been running on battery now for about 20 minutes. It says I have 25 minutes running time left. I bought that one new about six months prior to the SUA1500... so summer of 2009. I know it's history, so I'm sure it's still in much better shape. I'll be ordering the SUA1500 battery tonight or tomorrow.

When I had the computer off, I pulled out the Areca to try to get the cable off easier, and when I put it all back together, the battery was drained to 96%. That was about an hour total, probably. I know that's not the same as testing it like you said, during a read/write with the computer plugged into the wall (and unplugged) but I think I'm going to hold off until the new UPS battery comes in. I planned to replace that battery a long time ago, but kinda forgot about it. Heh.

 

[nanofrog]

I tugged, rocked, wiggled and everything for probably a half hour. It's in there good. I put it all back together because I heard some unsettling popping and snapping sounds as I tugged the crud out of it, and I wanted to make sure I didn't break something. It still works, but I've given up on it.

Yikes.

You did the right thing though IMO, as lead-free solder isn't all that forgiving in terms of flexibility. So if you flexed the PCB too much, you'd create bad solder joints, and the card would either cease functioning, or be flaky (i.e. act up when up to working temp, or some part that is only used rarely).

Even though you've already pulled the card as a means of removing the cable, it's your best chance IMO without damaging anything (also make sure you're not statically charged, as an Electro Static Discharge can be lethal to electronics).

Simply leaving the system plugged in (attached to earth ground), and touching the metal case before touching any electronics will make sure you're discharged before removing anything from the system.

For now, I pulled the UPS plug, and was alarmed to see the battery drop to 40% in about ten seconds. I think the big UPS (SUA1500) needs a new battery. I bought this used from a company that was going out of business for $25, and that was December 2009, so it's an old battery for sure. I pulled the smaller UPS (BackUPS RS 1500) that only has monitors and external drives hooked up, and it's been running on battery now for about 20 minutes. It says I have 25 minutes running time left. I bought that one new about six months prior to the SUA1500... so summer of 2009. I know it's history, so I'm sure it's still in much better shape. I'll be ordering the SUA1500 battery tonight or tomorrow.

The batteries definitely need changed.

Typically, you'll need to swap them out about every 3 years or so, though I've seen them last as long as 5 (don't bet on this long though, particularly with more recent batteries due to cost cutting).

When I had the computer off, I pulled out the Areca to try to get the cable off easier, and when I put it all back together, the battery was drained to 96%. That was about an hour total, probably. I know that's not the same as testing it like you said, during a read/write with the computer plugged into the wall (and unplugged) but I think I'm going to hold off until the new UPS battery comes in. I planned to replace that battery a long time ago, but kinda forgot about it. Heh.

I assume you're still talking about the UPS battery, and not the BBU for the card.

If it is the card's BBU that was down to 4% remaining in an hour, you probably got the wrong one (they introduced a new one for some of the 1880 series, which starts with the ARC-1880ix12 IIRC, as the larger port cards need more power = battery is rated for a higher current).

 

[wonderspark]

Yikes.

You did the right thing though IMO, as lead-free solder isn't all that forgiving in terms of flexibility. So if you flexed the PCB too much, you'd create bad solder joints, and the card would either cease functioning, or be flaky (i.e. act up when up to working temp, or some part that is only used rarely).

I would be very sad if I busted my card. I hope I didn't weaken any solder.

I assume you're still talking about the UPS battery, and not the BBU for the card.

If it is the card's BBU that was down to 4% remaining in an hour, you probably got the wrong one (they introduced a new one for some of the 1880 series, which starts with the ARC-1880ix12 IIRC, as the larger port cards need more power = battery is rated for a higher current).

No, I was talking about the BBU, and it was only down 4%. It still had 96% charge. I worded that poorly.

I did order a new SmartUPS battery from APC, even though other vendors had knockoffs for $100 less. I figure the free shipping on returning the old battery to APC is worth it for the reduced hassle and environment.

I also ordered the 8087-8088 cable. Once I get that, I'll see if it fixes the RAID3, then compare it to RAID5. I've found a couple of instances where people compared the same system in both formats, and RAID3 was about 100MB/sec faster every time, so I'm expecting the same, but it will be fun to test.

 

[wonderspark]

Well, got an answer about ports from Areca.

"Your Question : Is the 1880ix-12 using a total of 12 ports, or are there 16 ports when you include the 4 external ports via SFF-8088?

I'm connecting an external RAID tower via two cables... one is SFF-8087-SFF-8088 cable via internal port, and the second is SFF-8088 via external port. I've been initializing a RAID3 with six 2TB disks and one hot-spare, and it's been 38 hours so far with 95.2% complete. Is this normal, or have I got a conflict somewhere?

(Answer)
Dear Sir/Madam,

you can use all these external ports and internal ports at same time.
so you can say that it is 16 ports in total.

and initialization time may vary with drive, you can monitoring the drive
status of each array member drive to find out a possible problem. if there
have unstable drive or problem communication, the drive information page
will show you some errors in the error count.

Best Regards,

Kevin Wang

Areca Technology Tech-support Division
Tel : 886-2-87974060 Ext. 223
Fax : 886-2-87975970"

This makes me kinda bummed. I checked all seven drives in the RAID, and they all have zero error count, and say "normal" status. If this is all true, then the extra cable should not make any difference, and this is just the performance I'll be getting in this configuration.

Having spent 40 or so hours initializing, I'm hesitant to just delete it and build a RAID5 right away, with the chance that it won't be any better. I do have some work to do, so I think I'll leave it alone until the new internal cable arrives, and see if it makes any difference first.

I really appreciate all your help and attention with this. It's been very helpful!

 

[nanofrog]

NP.

You can share ports with multiple drives on these cards, so that's not an issue (will work without the need of additional hardware, such as a SAS expander). There is a performance penalty for doing so however (exact amount is dependent on a few different factors).

In your case, it won't be much in the current configuration, but testing out the disks on different ports for performance should reveal which ports are sharing bandwidth with the external port. Worst case, if you need additional performance, just add one of the existing Hot Spares to the set via Online Expansion.

Test it out (RAID 3), and see what you get, particularly during the failure conditions listed previously.

You can also look for online reviews (arecaraid.cineraid.com has had a good forum for this and thread on the 1880 series, but when I checked, the site is down).

 

[wonderspark]

In looking at other speed tests, I think I'm pretty happy after all. Take a look at these comparisons of my three main volumes.
One is just the original 640GB OS drive in slot 1 of the Mac Pro.
Next is the three original 1TB Hitachi/Apple drives in a RAID0, also internal (software) RAID.
Last is the RAID3 as is, with six 2TB drives.

Results 74.26
System Info
Xbench Version 1.3
System Version 10.6.8 (10K549)
Physical RAM 16384 MB
Model MacPro4,1
Drive Type Hitachi HDE721064SLA360 (640GBx1)
Disk Test 74.26
Sequential 149.84
Uncached Write 170.69 104.80 MB/sec [4K blocks]
Uncached Write 176.97 100.13 MB/sec [256K blocks]
Uncached Read 100.89 29.53 MB/sec [4K blocks]
Uncached Read 189.64 95.31 MB/sec [256K blocks]
Random 49.36
Uncached Write 17.30 1.83 MB/sec [4K blocks]
Uncached Write 158.71 50.81 MB/sec [256K blocks]
Uncached Read 93.79 0.66 MB/sec [4K blocks]
Uncached Read 159.42 29.58 MB/sec [256K blocks]
-----------------------
Results 189.26
System Info
Xbench Version 1.3
System Version 10.6.8 (10K549)
Physical RAM 16384 MB
Model MacPro4,1
Drive Type Internal RAID 0 3TB (3x1TB)
Disk Test 189.26
Sequential 250.86
Uncached Write 551.91 338.86 MB/sec [4K blocks]
Uncached Write 618.92 350.18 MB/sec [256K blocks]
Uncached Read 91.48 26.77 MB/sec [4K blocks]
Uncached Read 630.24 316.75 MB/sec [256K blocks]
Random 151.95
Uncached Write 71.48 7.57 MB/sec [4K blocks]
Uncached Write 508.63 162.83 MB/sec [256K blocks]
Uncached Read 169.70 1.20 MB/sec [4K blocks]
Uncached Read 223.51 41.47 MB/sec [256K blocks]
-------------------------
Results 271.20
System Info
Xbench Version 1.3
System Version 10.6.8 (10K549)
Physical RAM 16384 MB
Model MacPro4,1
Drive Type Areca Video (RAID3)
Disk Test 271.20
Sequential 896.46
Uncached Write 1361.15 835.73 MB/sec [4K blocks]
Uncached Write 2158.48 1221.27 MB/sec [256K blocks]
Uncached Read 353.44 103.44 MB/sec [4K blocks]
Uncached Read 2300.54 1156.23 MB/sec [256K blocks]
Random 159.77
Uncached Write 1533.41 162.33 MB/sec [4K blocks]
Uncached Write 42.14 13.49 MB/sec [256K blocks]
Uncached Read 2022.74 14.33 MB/sec [4K blocks]
Uncached Read 6278.27 1164.98 MB/sec [256K blocks]

What I'm seeing is that there are these consistently terrible numbers in the same areas of three different volumes, using Xbench. When I run the AJA test on all three, I'm seeing just a simple set of numbers (write / read) and they're pretty consistent.

AJA tests:
- 640GB single internal drive = 90MB/sec both write/read
- 3x1TB RAID 0 = 315MB/sec both write/read
- 6x2TB RAID 3 = 603MB/sec write, 550MB/sec read

Basically, I've done what I set out to do, which was greatly increase my capacity, speed AND reliability. I think the Xbench tests messed with my head somewhat, with those weird 4K numbers. Areca tech support has assured me twice now that the 1880ix-12 has 16 ports (that the 4 on the external are not sharing any differently than the 12 internal ports, as they all come from one chip.) So even though I'll definitely test for a difference with the second internal cable, I will still be happy if there is no change. I'd like to see it improve, but I think I'm in really good shape!

I'll also try the RAID 5, just to see how things go, and follow up on all of this.

 

[randomerratum]

Just thought I'd chime in-

I get pretty good performance on two systems running via a High Point Rocket Raid card... (1080 29.97 ProRes 4:22) except when we crash. If the Mac Pro crashes, it typically kicks the card and you have to get the RAID mounted with disk utility which takes multiple hours to come back online.

Crashes don't happen often, but when they do- it's usually when we can least afford a few hours downtime. We've actually moved over to our G-Raid FW800 backups just so freezes or crashes don't shut us down all day.

 

[wonderspark]

Just thought I'd chime in-

I get pretty good performance on two systems running via a High Point Rocket Raid card... (1080 29.97 ProRes 4:22) except when we crash. If the Mac Pro crashes, it typically kicks the card and you have to get the RAID mounted with disk utility which takes multiple hours to come back online.

Crashes don't happen often, but when they do- it's usually when we can least afford a few hours downtime. We've actually moved over to our G-Raid FW800 backups just so freezes or crashes don't shut us down all day.

Interesting...
What do you consider a crash? I'm trying to remember, and I want to say since December 2009 I had one, maybe two instances where Premiere crashed on me, but I don't think I've had the whole Mac crash yet. (Knock on wood!) I started with CS3, and since moved to CS5, heavily into After Effects and either 1080p or higher footage. (I was editing some crazy 4928x3264 timelapse video footage I created from over 70,000 still Nikon images, and Premiere choked on it. After Effects handled it great, however!)

I'm hoping I've covered my bases well in that regard. I have two UPS units for just this system; one is an APC SmartUPS 1500; the other is a smaller APC BackUPS RS 1500. The Areca RAID card has the battery back up installed as well. I've worked through several nasty lightning storms that knocked out power without a glitch so far. Here's hoping that trend continues!

What are you editing with when your crashed occur, may I ask? I've only just begun to use FCP X. (Not keen on it yet.)

 

[nanofrog]

Areca tech support has assured me twice now that the 1880ix-12 has 16 ports (that the 4 on the external are not sharing any differently than the 12 internal ports, as they all come from one chip.)

I'll also try the RAID 5, just to see how things go, and follow up on all of this.

What's thrown me, is when I look at their 1880 series cards, the 1880ix16 has 2x chips, not 1x, which is what I'd expect to be the case if there were 16x ports per controller chip.

As per performance between 3 and 5, 5 should give you another ~60MB/s or so than 3 under Normal conditions, assuming each of those disks is capable of 130MB/s or so.

I get pretty good performance on two systems running via a High Point Rocket Raid card... (1080 29.97 ProRes 4:22) except when we crash. If the Mac Pro crashes, it typically kicks the card and you have to get the RAID mounted with disk utility which takes multiple hours to come back online.

I'd be interested in elaboration as to what exactly is going on as well, as that sounds really odd (though Highpoint is the one company that doesn't surprise me, as they have a horrible reputation on the support end).

A software crash shouldn't take out the array, even if the OS is wiped out (assumes the OS is on a physically separate volume from the data).

 

[wonderspark]

What's thrown me, is when I look at their 1880 series cards, the 1880ix16 has 2x chips, not 1x, which is what I'd expect to be the case if there were 16x ports per controller chip.

Yeah, on this page, they show "16-28 ports" on the 12/16/24 models, and 8-16 ports on the i/x/LP and ixl8/12 models, so all the 12's have 16 ports according to the info... backed up by the emails I've received as well.

I misunderstood this myself, and thought 12 meant 12 ports, not 16. It's good to know that if I had to, I could add a second TR8X, though.

 

[nanofrog]

Yeah, on this page, they show "16-28 ports" on the 12/16/24 models, and 8-16 ports on the i/x/LP and ixl8/12 models, so all the 12's have 16 ports according to the info... backed up by the emails I've received as well.

Actually, I don't get that from the product page at all. They state connectors, but don't actually specify the port count (if shared or otherwise). It seems to me, they let the customer "interpret" that by inference the way I read it.

Now ideally, it's the stated port count + 4 on those with ix in the model number. But the chips aren't designed that way.

For example, if their chips actually contain 16 ports, then a 24 port internal card (ARC-1880ix24) would actually contain 32 ports, not 28. Now if this is the case, they could have helped users with an additional SFF-8088 port, and the PCI bracket should have sufficient physical space to fit it (tight, but it would fit if they shifted the DIMM slot on a slightly longer board, which is possible as it's not at the max length spec at it's current size).

Given the email, 16 ports per chip is likely the case, and users could get say an ARC1880ix16, flash it with the ARC1880ix24 port firmware, and add in the additional SFF-8087 connectors if they have right skills and equipment to solder the components (it's the same exact PCB).

 

[wonderspark]

Actually, I don't get that from the product page at all. They state connectors, but don't actually specify the port count (if shared or otherwise). It seems to me, they let the customer "interpret" that by inference the way I read it.

Now ideally, it's the stated port count + 4 on those with ix in the model number. But the chips aren't designed that way.

For example, if their chips actually contain 16 ports, then a 24 port internal card (ARC-1880ix24) would actually contain 32 ports, not 28. Now if this is the case, they could have helped users with an additional SFF-8088 port, and the PCI bracket should have sufficient physical space to fit it (tight, but it would fit if they shifted the DIMM slot on a slightly longer board, which is possible as it's not at the max length spec at it's current size).

Given the email, 16 ports per chip is likely the case, and users could get say an ARC1880ix16, flash it with the ARC1880ix24 port firmware, and add in the additional SFF-8087 connectors if they have right skills and equipment to solder the components (it's the same exact PCB).

Haha, you know, I considered that as I was trying to unplug that stupid cable (which I never did)... if I broke the internal connector, I might be able to resolder a new one into one of the three empty spaces. I have the skills, and a soldering iron somewhere around here.

I agree that the specs are not 100% clear. In a second email to Areca, I asked if there are 16 independent, unshared ports on the 1880ix-12, and the reply was:

"How you define sharing? And how you define independent? All ports came from same expander chip but every port is standalone. I do not know it mean sharing or independing [sic] for you.

Best regards,
Kevin Wang"

So from that, I take it to mean that if there is any sharing, it's sharing evenly across the 16 ports. If you use 8 ports, they are used equally, regardless of where you plug into them. I think they should call it an 1880ix-16/20/28 instead, just to make that clear.

 

[nanofrog]

Haha, you know, I considered that as I was trying to unplug that stupid cable (which I never did)... if I broke the internal connector, I might be able to resolder a new one into one of the three empty spaces. I have the skills, and a soldering iron somewhere around here.

I agree that the specs are not 100% clear. In a second email to Areca, I asked if there are 16 independent, unshared ports on the 1880ix-12, and the reply was:

"How you define sharing? And how you define independent? All ports came from same expander chip but every port is standalone. I do not know it mean sharing or independing [sic] for you.

Best regards,
Kevin Wang"

So from that, I take it to mean that if there is any sharing, it's sharing evenly across the 16 ports. If you use 8 ports, they are used equally, regardless of where you plug into them. I think they should call it an 1880ix-16/20/28 instead, just to make that clear.

As per soldering, you'd need hot air or an oven, not an iron. And a broken connector would have to be replaced with it's existing firmware.

What a nice, concise and clear answer...

Ultimately it works, and works well (going off of the performance data), so I wouldn't worry about it.

 

[wonderspark]

Update with two internal cables vs. one internal, one external:
AJA test @ 1920x1080 10-bit:

- 6x2TB RAID 3 = 625MB/sec write, 551MB/sec read

So, slight improvement, but probably just coincidence, not actually faster.

 

[nanofrog]

Update with two internal cables vs. one internal, one external:
AJA test @ 1920x1080 10-bit:

- 6x2TB RAID 3 = 625MB/sec write, 551MB/sec read

So, slight improvement, but probably just coincidence, not actually faster.

Cache Disabled or Enabled?

 

[wonderspark]

Cache Disabled or Enabled?

Enabled. The write speed bounces up and down, apparently, but here are screenshots of the same test back-to-back just moments ago...

 

[nanofrog]

Enabled. The write speed bounces up and down, apparently, but here are screenshots of the same test back-to-back just moments ago...

They do vary, but it's a bit odd that the write values went up with cache disabled, even if wasn't by much.

 

[wonderspark]

They do vary, but it's a bit odd that the write values went up with cache disabled, even if wasn't by much.

I picked the 16GB test to get the longest run. It also depends on if I'm still running Firefox, I think. If I do the 8GB test, the read numbers are up in the 2000s or so, and the writes are in the 700s... watch this, I'll do it and post up...

 

[wonderspark]

Here's the same test with 8GB files instead of 16GB.

 

[nanofrog]

Here's the same test with 8GB files instead of 16GB.

Reads are way off on the right hand screen shot...