BIG storage without breaking the bank...

[nuno1959]

hi everyone !
computer ''newb'' here & have decided to put my music/photos/video onto HDD.
i have 10.000 albums ( 35 years of collecting CD's & LP's, plus 400 DVD's ..) & started fiddling around with some W.D. My Book USB drives to ''test the waters'' : so far i have 2 X 1TB drives plus 1 x 2TB drive almost full & haven't ''scratched the surface'' of it ....but am happy w/ the results !
transferring my music in uncompressed/lossless formats - i realize i need a massive capacity external storage unit, complete with back-ups + ''unplugged/stashed elsewhere'' further back-ups for it to be SAFE ( redundancy ?.. )
not having endless funds & being a silversmith w/ excellent fabricating skills & all necessary tools, i'm ready to build something to keep it all, my idea being :
for iTunes/music - 8TB+back-ups, for iPhoto/pics - 2TB+back-ups, for my video - 4TB+back-ups : in one box, two boxes , several ??
googled up the subject & i'm overwhelmed by the vast amount of info & how it branches into related matters : RAID 1, 2, 5, 6, 10 /non RAID, NAS, JBOD, concatenated, spanned, the list is endless !! still, i'm decided & would like to retain the W.D. plug'n play/no fuss simplicity, coupled with the big capacity i need, my computer being a 1 year old alu iMac, i guess either USB or FireWire would do - the My Books play flawlessly through plain USB ...
my ''cry for help'' is that i'ld REALLY need someone to provide me like a parts list of compatible/proven components or point me in the right direction ( a ''how to'' book, DIY Kits, manufacturer site ?.. ) to get the info needed/job done - so far all i've come up with are commercially built NAS boxes people seem to consider not ''so hot'' for what they cost, home built NAS boxes, up to 2, 3, maximum 4TB capacity, but apparently not Mac compatible...
i AM feeling a bit lost here & ANY help would be GREATLY appreciated, thanks in advance
best wishes
Nuno

 

[jampat]

A NAS by definition should be platform independent. The NAS controls its local drives and appears on your network. GigE is more than fast enough for general storage requirements. A homebuilt NAS will likely run on Unix and normally requires equal sized drives (assuming you want RAID5 for a good balance of speed and redundancy).

A Drobo does cost a lot more up front, but allows you to swap out one drive at a time with a higher capacity drive as required. This could really come in handy as your collection expands.

If you talking about 14GB+redundancy, budget will not be low. You will be looking at at least $2000 in hard drives plus the controllers. As you will be investing a lot of time and energy in this array, I would create two entirely separate arrays in different locations. Otherwise one fire or theft could wipe out all of your hard work.

Make sure you test out your ripping methods early on in the project and are happy with the results. You don't want to rip 10,000 albums and realize the tags weren't setup properly or your software didn't deal well with scratched CD's.

Sorry I couldn't give you an easy answer, I am in a similar place, but with much lower storage requirements.

 

[ChrisA]

It i pretty clear that you need a RAID system. RAID-5 is the way to go. A RAID system this large will have quite a few disk drives and will almost certainly be rack mounted. It will not be cheap. The raw disk drives will cost about $100 per terrabyte and expect to pay that much or more for the box they plug into.

You can buy a ready made storeage system or build your own server using a rack mount PC chassis, UNIX and several rack mount disk arrays.

There are two ways people use to back up these thinks (1) tape cartridges or (2) buy identical RAID systems.

Our company keeps three large RAID system each in a different city and they sync these continously using high speed network connection. in your case the data will not change so after you build the clone of your big RAID system you can update it without the a high speed connection. But in any case the only practical way to off-site backup a 24TB RAID is to have an identical RAID at a remote location. You are looking at _at least_ $12K for this project if you can do most of the work yoursef. Double of tripple that if you have to buy a turn key system.

You are WAY past the point where a few "my book" external drives can be made to work. For example when they say a drive has an expected life of 36 months then if you own 36 drives you can expect one to fail every month. Your system must be able to handle monthly failures and your budget will have to be able to handle the monthly cost to replace broken parts, not just drives but power supplies and so on.

 

[mrwizardno2]

I run an 8 core opensolaris server that I built with an asus mobo that has 12 sata ports. No fancy disk controller needed. ZFS handles all of the goofiness, and once it's set up it's a piece of cake. In your case, you may need an additional controller, but if you used 2tb disks, I doubt it.

My setup is about 10TB raw disk, but about 8 useable because of my zpool scheme. It'll be interesting in hearing what you end up doing! If I were you, I'd definitely take a good hard look at ZFS. While Solaris might not be something you can buy and just use - it certainly isn't hard to set up. Once you're done, you likely won't have to fiddle with it again unless you want to add more storage.

 

[PinkyMacGodess]

You could try something like this from HP. Or even this one again from HP.

Do it yourself is one possibility from these guys here and here too.

We did have a client that was sold on the idea of disk to disk backup and using a dual channel disk array for their main systems. Well, until they got the proposals. Even doing it 'on the cheap' was very expensive...

 

[RedTomato]

I would honestly rethink this whole thing.

HD storage goes down in price by 30-50% per gigabyte per year. If you wait a few years, you will be able to get your entire collection onto a few cheap HD drives.

Of course, you don't want to wait a few years. I would focus for now on picking out your 100 or 200 or similar favourite items of media and building a maybe 3TB raid-5 system (4 x 1TB drives) to store them all.

Big storage arrays are complex as you've learned, and it's easy to make expensive mistakes. Start with a small system, maybe with cheaper 1TB or 1.5TB drives, not 2TB drives, to learn the ropes. Just learning about this then filling this up will take you the best part of a few months, and by then 2 TB drives will be cheap, and 3 TB drives will be on the way.

To hold your learner storage system, I suggest finding / buying a cheap tower or midi-tower case with space for maybe 8 drives, and a cheapish quad-core processor, then you can experiment with:

- raid cards / software raid / dual gigabit ethernet ports (either on the mobo or via add-in cards) etc.

- library / archival / retrival / jukebox systems

- format shifting / lossless compression

- mounting 3 or 4 or more optical drives to ease the chore of uploading 10,000 cds / dvds one by one.

Test this out by using it to i) transfer from cd / dvd ii) store and iii) watch/ listen to maybe 50 of your favourite items. You'll find there's a lot of small things that need to be improved on, before going ahead with creating a big system which is what this test system is there for.

As your skills improve, you can throw in more drives. Don't forget dealing with enlarging RAID systems / replacing failed drives is a whole new field where there are lots of mistakes waiting for you to make them.

Make test RAID systems, add disks and unplug them to stimulate failures. Expect to lose all the data on your RAID system a few times till you've learned the ins and outs of looking after them.

Big storage racks - but don't think about getting one just yet:

http://www.wegotserved.com/2009/06/11/hands-on-norco-rpc-4220-rackmount-chassis/

That one's off the shelf, but there's another online-storage company that home-made their own massive storage units, and published the blueprints. They were reviewed quite widely on the tech sites.

This would be more like what you're looking for, but I can't find it with a quick google.

Hope that helps.

EDIT: Still not the link I was looking for, but here is quite an old guide to building your own rack. Personally I'd just go out and buy a rack.

http://www.hauntedyards.com/images/books/rackbuildinghowto.pdf

EDIT II:

Ah! Finally found it!!! This is what you're looking at with current tech. 67TB in a single home-made 4U chassis for about $7000.

Note that there are serious issues with their design. If you're gonna copy their example for your own use, you'll find it works well for them, but not for for you. But it should be an excellent pointer for you to create something that matches your needs.

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

x RedTomato

 

[nuno1959]

hi guys !
Thanks for the input & forgive my tardiness to reply - didn't get notices of answers : have to check my profile settings...
even if i ''don't want to break the bank'', i'm fully aware this will cost me some serious €. then again if i consider the worth of my music collection + countless hours/amount of work getting it to iTunes versus the cost of any of these storage solutions, i'd think it's money well spent.
someone suggested i use a Mac Mini connected to my iMac, with as many drives as possible ( like these ? : http://www.fnac.pt/pt/Catalog/Detai...pd_stockDisponivel&template=Template Generico .. ) connected through these for example : http://www.fnac.pt/pt/Catalog/Detai...=Hardware&category=hubs&product=5099206003828 - i have two of them & they work FLAWLESSLY ! this would potentially allow me to use up to 20 of these drives !? have to make sure iTunes can read from multiple drives...
then to round it all up, have the exact same drives, unplugged/elsewhere for safekeeping & redundancy's sake...
i know the whole thing reeks of ''totally over the top'', but having done a few dozen LP's into iTunes ( not too good, have to re do some.. ) i think i'd shoot myself in the foot if i'd have to re do them ALL again because of drives going bad/no redundancy.....hence my need to take AS MANY PRECAUTIONS AS FEASIBLE !! that , coupled with my ignorance on the matter at hand helps a lot, of course...
mrwizardno2, i'd be VERY interested in knowing more details about your set-up, not being a free-loader wanting to benefit from your hard work, i'm sure we could work something out !?
i do WANT to make this happen, in whatever form - the My Books ''experiment'' have shown me a world of possibilities that is both very exciting in it's simplicity, convenient + the MASSIVE amount of space saved !?? & the iMac as a media center would make a perfect set-up : it's quiet - never heard it's fans.. - simple to operate, stable. as is, i have it connected to my plasma screen + my hi-fi & it works an absolute treat : the lot controlled by an universal learning remote ( Front Row included if you please.. ) & w/ my wireless key/mouse sometimes i even check my email from the couch !! if this won't redefine the term ''couch potato'' what will ?...LOL...
please do keep those suggestions coming, O.K.? i'm sure something pretty amazing could pop-up from this discussion...
much obliged
Nuno

P.S. : Hi Red Tomato - this is like an afterthought 'cause i only noticed your reply AFTER i posted mine....OOOPS ! sorry !...LOL...
you do offer useful/sensible suggestions in your post but - call me lazy if you want... - it's the experimenting bit i want to steer clear off. i'm just not knowledgeable enough technically, just to give you an idea of how ignorant i am, the other day i moved my whole iTunes from the 1TB drive to the 2TB drive by means of ''click'n drag'' thinking that would be that right ? & i still haven't sorted why i don't ''have it'' - even if i know i haven't ''lost my music'' !!...let's just say i'm not the brightest of the bunch & leave it at that ....LOL, LOL..
oh man ! digital life IS sweet but...it should be easier.. ;-)

 

[RedTomato]

Hm, you are certainly a music lover and probably a music expert too, but it seems you are less expert in computer issues. I think we all missed the bit where you say you're a computer novice:

hi everyone ! computer ''newb'' here

Many of the suggestions in this thread will certainly be too complex for you to set up and maintain.

Setting up is the easy part, but the real time killer will be in importing from CD and DVD. Over time, there will be a big hassle and expense in maintenance and keeping it all working as technology and formats change.

Even a small error could kill the whole thing, e.g. a corrupt database that then gets copied to your backup, killing the database there too.

To be honest, it seems you have more important things in your life than learning how to create and look after an archive of 10,000 CDs and DVDs.

I would scale back your ideas to just putting on disk your favourites, rather than every single CD / DVD you own.

Jampat mentioned a Drobo. This seems about right for you - an 8 drive Drobo is expensive to get but it will do a lot of the techy stuff for you. Put two or three 2TB drives in, then when you fill them up, put in another one and so on.

I'm not sure if you can stream a DVD or Blu Ray from a Drobo, but other people can answer that.

There's a possibility that a new update will come early next year with USB3, which is one reason to wait, if this isn't for your work. If this is for work, then get what you need now.

Hope that helps.

 

[ChrisA]

Hm, you are certainly a music lover and probably a music expert too, but it seems you are less expert in computer issues. I think we all missed the bit where you say you're a computer novice:

I have to agree with this. When I think of the disaster that hooking up 20 or so "My Book" style external drives would create. The problem is handling the all the failures. And YES if you own 20 external disks you can expect the kind of failue that totally corrupts a database to happen several times a year, maybe even monthly. It would take days to recover each.

What you need is a system that degrades gracefully when stuff breaks. RAID-5 can do that. You are going to have to buy a disk array that implements RAID5. There are many Drobo is just one of them.

If it were me setting this up I'd use a Solaris server and put the data on a ZFS file system. But this is not for a computer novic. You are going to need a turn-key RAID solution. Using a bazillion external drives is no a good solution

Backup is hard. Having just one identical RAID5 disk array is a minimal solution. The rule of thumb is to always have three copies of the data. OK if we count the CDs and DVDs you have in storage then that makes three.

To count as a backup the media MUST be in a different geographical location at all times, even during the backup. As soon as your only backup copy is physically collocated with the on-line copy then you are only one power surge from loosing both copies. Lightening hitting a utility pole many miles away can cause equipment failure in multiple drives. Be sure to budget for a redundant, un-interruptible power supply and power conditioning. Then look at the heat load. You will be burning a few watts of power and will need to remove the heat from the room

The "drobo pro" is looking very good. I'd say it's ideal. Buy a small rack. mount a APC "smart UPS" and one 8-drive drobo and leave space for a second drobo
http://www.drobo.com/products/drobopro/index.php

 

[nuno1959]

hi everyone !
RedT. & ChrisA. : you're breaking my heart guys !!...LOL...however you're both right about my (in)ability to deal w/ it, should IT go wrong..
i always loved/hated computers until 2 years ago, a cousin of mine let me fool around his new 20'' Intel iMac & i fell for it's unassuming, apparent simplicity. bought 1 & little by little got drawn into all sorts of stuff, photography for example : used to hate the damn thing but between a digital camera/iPhoto's ease of use, now i can't stop ''shooting'' !...LOL... = 4000 shots during my recent 14 day trip to the Outer Hebrides
music came naturally in the ''experimenting frenzy'' : i do have a nice music collection which i LOVE to bits, i'm an artist/work at home & i listen to music like....14 hours a day !?...LOL... i breathe/need music, it unleashes my own creativity, joy, you name it !!....as i told you, i experimented with the My Books & them being so fuss free, i assumed that + HDD TB's would be more of the same simplicity.....
by now i can see it isn't - damn shame..
having sorted computer versus Hi-Fi sound quality issues ( in my case a cheap & merry USB DacMagic by Cambridge Audio, unreal sonic quality for ANY price.. ) i was really looking forward getting rid of CD & LP handling + the cabinets to store them, the decks.....i was wishing the future before it's actually here, at least in a ''fit & forget'' manner.
i did look at the Drobos & liked them - learning about them is what ''launched'' me on this quest. i didn't fall for them however because of one detail that put me off : it's their own proprietary software/process/whatever of back-up technology & that sent a bell ringing in my head : if something does go wrong ( good old Murphy.. ) they'll be the ONLY ones capable of ??...know what i mean ? maybe it's not THAT important after all...
so...for the time being i'll keep on fiddling/learning the ropes with the My Books & see what does come up in the next few months, my main purpose being to yes, go digital but mostly DO IT RIGHT ! once & then forget about it...
anyway : thank you VERY much for all the suggestions you all brought forth, i AM learning a whole lot out of this
N