Sidebar Sidebar
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.

I need some help. I have 20 TBs worth of storage; and I need more. Need help!!

WiiDSmoker

WiiDSmoker

Suspended
Original poster
Sep 15, 2009
1,891
7,431
Dallas, TX
Currently I have 20 Terabytes worth of storage. 10 2 Terabyte drives. They are all completely full. I'm estimating that I need about 35-40 more Terabytes to complete my collection. Unfortunately I haven't even been able to begin to back things up; which I'm worried about, but not currently at this time.

I've contemplated running a bunch of 5 Bay NAS's; but I'm thinking I'd like to try something else.

Is there anyway I can get a long box that can work like a NAS; but hold 10-20 SATA drives. Or something!

I need some ideas. Please help :(
 
Something like this?

41WMuE4MXQL._SS400_.jpg
 
As an Amazon Associate, MacRumors earns a commission from qualifying purchases made through links in this post.
I required more storage so I built an 8Bay freenas box, specs are in my sig and it cost me around $700. Your cheapest option is going to be finding a PC case (most likely going to be a rack mount server case) that can hold more than 10 drives, build a PC inside of it and throw freenas on it.

Not only will you need a case that can support the drives, but you'll need a raid card as well. I bought a Highpoint card, which I love, so I'll recommend it to you as well. I know they have a bunch of cards that support more than 10 drives, so check them out.

Keep in mind I'm assuming that someone that requires 50TB of storage is also somewhat technically minded. Picking out parts and building a custom NAS server can be pretty tricky so if you have no idea what I'm talking about make sure you do a lot of research.
 
WiiDSmoker said:
Currently I have 20 Terabytes worth of storage. 10 2 Terabyte drives. They are all completely full. I'm estimating that I need about 35-40 more Terabytes to complete my collection. Unfortunately I haven't even been able to begin to back things up; which I'm worried about, but not currently at this time.

I've contemplated running a bunch of 5 Bay NAS's; but I'm thinking I'd like to try something else.

Is there anyway I can get a long box that can work like a NAS; but hold 10-20 SATA drives. Or something!

I need some ideas. Please help :(
Click to expand...

You will need cash... even the freenas way is expensive. I would say it's almost impossible to do better than just buying off the shelve. The reasons why it will be hard/expensive.:

- You will need a special controller PCI card with the extra connectors. These are expensive.
- You will need to buy special server HDD's. It can be done with consumer grade HDD's, but it's playing with fire (literally , it gets HOT)
- You will need a very big, well cooled case.
- You need a server grade PSU (actually you want two PSU's, it's a redundancy factor)
- 40 TB means that you will need approx 60TB just to create a Raid array... and that's still not a backup. I would definitely recommend at least raid 5, preferably raid 6. The reason is that even if you backup this array you want some redundancy (with these amounts of data you probably won't have a completely real-time backup, the reason is in the following post).
- You want two arrays... and preferably the second one off-site (a data-center? Family? Friends?). Loosing so much data is just inconceivable, so you want to create a backup, hence the second array. Off-site is the safest way, just do an incremental sync at night (this is why you might not have a completely synced backup during the day). If it's in your house you can just do a 1-1 sync... but you are not disaster proof. Pick your poison.
- You need to have a powersupply that can cater that many SATA power connectors. Once again you want to have a server grade powersupply.
- You need to setup freenas so you can sync with ISCSI or off-site. This might be challenging, though there will be people who have done this.

To be honest: unless you are really a techie who loves to tinker... not a very practical solution. The realistic option would be to buy a two 8U 19" racks and host one of it at home and one of it in a data-center. But be ready to fork out 10K for each array......
 
This may not answer your question at all, but what exactly is filling up all that space? I have a feeling it will be full DVDs and Blu-Rays.

In cases like this, it almost makes more sense to just rip the stuff out of those files that you don't need. The savings can be huge (although the time you will spend will also be huge).

Most big budget DVDs have several other languages and special features you will never watch. Save the full DVDs for movies you don't want to lose anything for, but just save the movie for other films.

But yeah, that amount of storage is currently in the realm of, "you will need to figure it out yourself" and no easy solution exists. Sorry kid.
 
Sounds like you need help in the same way the people on Hoarders need help. I have 2 TB on my iTunes server, and that is holding 670+ movies and about 600 TV shows. If you are needing like 60TB, you are talking about thousands upon thousands of movies. 10,000+. If you are buying these legally, you are looking at well over $100,000 worth of movies, so you would apparently have enough money to buy whatever kind of storage you think you need. If you aren't filling these TB's legally, (which I'm pretty certain is the case) I'm going to keep my eye on the news waiting for the largest piracy case in history.
 
iowamensan said:
Sounds like you need help in the same way the people on Hoarders need help. I have 2 TB on my iTunes server, and that is holding 670+ movies and about 600 TV shows. If you are needing like 60TB, you are talking about thousands upon thousands of movies. 10,000+. If you are buying these legally, you are looking at well over $100,000 worth of movies, so you would apparently have enough money to buy whatever kind of storage you think you need. If you aren't filling these TB's legally, (which I'm pretty certain is the case) I'm going to keep my eye on the news waiting for the largest piracy case in history.
Click to expand...

I would agree that, regardless of how you got the movies, you are keeping 90% of what you will probably never watch again. Every now and again I go through my 5TB collection of movies and shows (which is pretty epic in my opinion) and prune out the shows that I thought I wanted to keep but really don't.
 
I was all set to come on here and stick up for the OP by saying that you shouldn't make assumptions about what is filling up his harddrives. It could well be that he is into digital video creation which can fill up terabytes fast, or be a photographer with tons of raw photos. Whatever, the point is that it is possible that these aren't pirated movies.....

But then I saw that his username was WiiDSmoker :rolleyes:
 
WiiDSmoker said:
...
Is there anyway I can get a long box that can work like a NAS; but hold 10-20 SATA drives. Or something!

I need some ideas. Please help :(
Click to expand...

Assuming that you really do need all this data and that your need for storage will continue to grow over the coming years. You are just going to have to make the jump to a professional level storage system. I'm very partial to Solaris based servers because of their ZFS file system. But if you'd rather not go that way because of the learning curve assoicated with Solaris you can use Linux. Either way you end of with a NAS system that has the performance to steam many real time HD video streams oover gigabyte Eithernet cable(s)

You will need a high performance server and a robust stagage system and a large UPS and a rack tomount it all in. You might need to think about cooling but it will not really be that big of an issue onless you put the rack on a closet that is sealed.

We use EMC/Clarion here. They are cost effective and reliable. But do configure RAID and multiple hot spare drives. The system will fail over to the spares and you can replace the failed drives whenever you get around to it



http://www.emc.com/products/launch/clariion-ax4/index.htm
Prices are very roughly on order of $1K per TB of configured starage. Yes, higher then the price of bare disk drives.
 
OllyW said:
Something like this?

41WMuE4MXQL._SS400_.jpg
Click to expand...

I'm actually looking into the Norco 4220. It's only $299 and it can hold 20 Hot-Swappable SATA/SAS Drives.

iowamensan said:
Sounds like you need help in the same way the people on Hoarders need help. I have 2 TB on my iTunes server, and that is holding 670+ movies and about 600 TV shows. If you are needing like 60TB, you are talking about thousands upon thousands of movies. 10,000+. If you are buying these legally, you are looking at well over $100,000 worth of movies, so you would apparently have enough money to buy whatever kind of storage you think you need. If you aren't filling these TB's legally, (which I'm pretty certain is the case) I'm going to keep my eye on the news waiting for the largest piracy case in history.
Click to expand...

All my DVDs and Blu-Rays are 100% legit; store bought. The problem I have, is that I want to keep all movies in their ISO format as well as encoded into mkv/m4v.

emvath said:
I was all set to come on here and stick up for the OP by saying that you shouldn't make assumptions about what is filling up his harddrives. It could well be that he is into digital video creation which can fill up terabytes fast, or be a photographer with tons of raw photos. Whatever, the point is that it is possible that these aren't pirated movies.....

But then I saw that his username was WiiDSmoker :rolleyes:
Click to expand...

Nope. Just 100% movie rips. WiiDSmoker is a play on words. Nintendo Wii. Nintendo DS. And of course weed and smoker :p


I'm looking into getting an Atom based Motherboard with a 560 watt 45-50 amp Powersupply. And then I would need to get PCI Express SAS/SATA Raid cards.
 
As an Amazon Associate, MacRumors earns a commission from qualifying purchases made through links in this post.
A couple of unraids (perhaps using that Norco box(es))? That can be fairly cheap compared to other solutions listed here, and is pretty good for media storage. Expand it as you need it without rebuilds.

Also, if the OP owns all those discs, why keep an ISO copy too? Why not dramatically reduce his storage requirements with the MP4/M4V versions, and just go get the disc when he wants the rest?

Lastly, if he does own that many discs, money is not really an object. If that's the case, maybe unraid (the poor man's big server) is not the way to go. Just spend big and buy big and best.
 
WiiDSmoker said:
I'm actually looking into the Norco 4220. It's only $299 and it can hold 20 Hot-Swappable SATA/SAS Drives.

I'm looking into getting an Atom based Motherboard with a 560 watt 45-50 amp Powersupply. And then I would need to get PCI Express SAS/SATA Raid cards.
Click to expand...

Just for some comparison, I have a norco 2208, a 500w psu, and an i3 2.93ghz. Although I havent really come close to maxing out the i3, I would let an Atom run that big of a storage array. I have the Highpoint 2720 card which supports up to 8 6GB/s SAS/SATA drives but you'd need to either get a few of them, or just one card that supports 20 drives. Check out the 2760
 
Lot's of case options available to you if you search around - certainly doesn't need to cost silly money.

I think solutions like freenas/nexentastor/unraid work just as well (better) as the expensive commercial solutions - they just come with less (no) support and aren't bought by an accountant through an enterprise contract - way to generalize :)

Personally, I would use (and do) go for a bsd/solaris based machine and use zfs - which also avoids the huge expense of a raid card.
 
reebzor said:
Just for some comparison, I have a norco 2208, a 500w psu, and an i3 2.93ghz. Although I havent really come close to maxing out the i3, I would let an Atom run that big of a storage array. I have the Highpoint 2720 card which supports up to 8 6GB/s SAS/SATA drives but you'd need to either get a few of them, or just one card that supports 20 drives. Check out the 2760
Click to expand...

Thanks for the info man. That card looks awesome. May I ask what motherboard you are using for that i3? Is it onboard video?
 
You're into the realms of needing a tape management system and tape robot. You allocate so much space for permanent storage of files and another large area for stuff to be restored from tape. Either with a strict retention policy, ie files returned from tape are only kept on disk for 24 hours / 7 days, or when a file is restored to disk, it overwrites the oldest files on the disk.
A tape robot and tape management system should be readily available on the 2nd user market, if you can get them before they go to India / China.
The big firms just lease them from the big suppliers.
 
WiiDSmoker said:
All my DVDs and Blu-Rays are 100% legit; store bought. The problem I have, is that I want to keep all movies in their ISO format as well as encoded into mkv/m4v.
Click to expand...

problematic. what mediums do you play your movies on? computer connected to TV? etc.... maybe there is a work around.

assuming you have the money, you would setup your main RAID system - preferably a parity based RAID, RAID5 if you want speed, RAID6 if you want redundancy.

had you considered online backups? Amazon or similar? they charge $0.15 per GB. 50TB would cost $7500! holy smokes haha!
 
WiiDSmoker said:
Thanks for the info man. That card looks awesome. May I ask what motherboard you are using for that i3? Is it onboard video?
Click to expand...

Just a cheap ECS H55H-CM board. I do use the onboard video but 99.9% of the time I access the server from the webconsole. If FreeNAS intrests you check out their forum or you can PM me if you have more specific questions.
 
WiiDSmoker said:
I'm actually looking into the Norco 4220. It's only $299 and it can hold 20 Hot-Swappable SATA/SAS Drives.



All my DVDs and Blu-Rays are 100% legit; store bought. The problem I have, is that I want to keep all movies in their ISO format as well as encoded into mkv/m4v.



Nope. Just 100% movie rips. WiiDSmoker is a play on words. Nintendo Wii. Nintendo DS. And of course weed and smoker :p


I'm looking into getting an Atom based Motherboard with a 560 watt 45-50 amp Powersupply. And then I would need to get PCI Express SAS/SATA Raid cards.
Click to expand...

Atom based server??? Most low cost RAID cards are actually software RAID. The CPU will need to be able to handle RAID calculations and stream the data out the network and possible re-build a faild drive all at the same time. Most people who build these systems are using at least dusl CPU systems. YOu really don't want t a low end CPU. Think about backups it will take hours or even days to back this think up and alll that time the CPU is "cooking" and I assume you'd still like to be able to access you data while the backup goes on.

Next think about software. What file system will you use, volume managment. You will nedd a battery backup system. All those drives have 16 or 32 MB of RAM cache and it is lost when power fails with 20 drives you can have 1/3 GB is RAM at any one time.

It should be obvious we are talking about a 64 bit CPU and 64-bit OS
 
Sounds like your first step is to set a budget and then decide if you want a do-it-yourself option like FreeNAS or you want a commercial solution. Either way it sounds like a fun project and I hope you'll keep us updated as you go!

If you want to go the NAS route you can definitely do better then a bunch of 5 disk units. Here is one example of a 12 disk NAS:
http://www.readynas.com/?cat=73

.
 
Devil's advocate:

You are keeping the ISO versions of your discs as a backup I am guessing, but surely the DVD and BluRay medium you ripped them from is a MUCH more stable medium than hard disc?

Either go the tape route someone mentioned and just put those ISO files on them and store them off your local drives or just be like most other sane people and get rid of the duplicate ISO files. 40TB is an insane amount of storage space, especially when 80% or more of it is something that you really are not even going to be using again.

I'm sure in 15 years or so computers will have 40TB of storage standard, but right now it's an overwhelming and unnecessary amount given your situation.

If you have the money and patience for it, knock your socks off I suppose, but you really should rethink your methods...especially since you haven't even been able to back up your drives up to this point. I assume you are keeping the ISO files in case something happens to the originals. Heck, you could lose all of that in an instant and STILL have to go back to your original source discs if the drives fail.
 
ChrisA said:
Most people who build these systems are using at least dusl CPU systems.
Click to expand...

Personally, I've come to the conclusion that something like a discounted low-end HP DL180 from a VAR starts to quickly make more sense than a scratchbuild.
 
You must log in or register to reply here.
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.
Top Bottom
Back