Lots of disks for media collection

[Ryan P]

It wasn't quite my original intention, but my Mac Pro has turned into the center of a new home theater I've set up with Plex and a 1080p projector. I've been ripping bluray movies to disk, downloading from itunes, etc, etc and I am really struggling with disk issues. My photography and limited video work were nothing compared to this.

Right now I've got about 7 TB's of Media files and I don't see this addiction stopping anytime soon. I have a mixture of drives in my 2010 Hexcore that is maxed out and 1 drive literally hanging out the back on a SATA card.

I've been waiting for the WD 3TB Black to come out to do upgrades to my internal drives. I'd like to at least get my internal drives fast, as that does seem to be a bottleneck for many things I do. (But yes I get you don't need fast drives for playing back movies!).

I've been playing with one of those Synology NAS boxes at work and that seems an interesting option although the price per GB is a bit more than I would like. I'm also worried about the noise from all the drives running, but they seem to support the 3TB Green's with the new 1511+.

My latest thought is to actually go USB 3, with 2 USB 3 hubs and perhaps 8 2TB Lacie drives. With Plex, I can still map them all to the same media directory and that way only the drive I am accessing is spinning up. Then internally I can go with the 3TB WD Blacks when they ship.

http://www.bhphotovideo.com/c/product/745553-REG/LaCie_131102KUA_USB_3_0_HUB.html
http://www.bhphotovideo.com/c/product/737354-REG/LaCie_301967_Minimus_USB_3_0_2TB.html

I'd rather a 3TB USB 3 drive but those Lacie 2TB's look like they would stack and not take up much space at all.

I guess it would be cheaper to not try and have it all online, but this seems not too much more expensive and a lot more convenient.

Thoughts? That should get me up to around 30 Terabytes. I'm somewhat thinking for backup plan is simply to just rerip media for now. I do offsite backups for my photography, but doing that for 30 Terabytes of Media files seems too much!

 

[EtherealMAC]

Here's my suggestion:

Definetely the most straightforward soution is as you say, to replace the internal drives with 3TB ones. However, before you consider investing in a usb 3.0 card and in 3.0 external hardrives, I think you should consider a more obvious and practical solution : buy firewire 800 external drive enclosures, reuse your old drives by putting them in these enclosures and then just DAISY CHAIN them.

Why you say? Well, first of all usb 3.0 is fast an all, but guess what, Lightpeak is faster. Waaay faster. I give it a year or so for lightpeak desktop cards , and lightpeak external hard drive enclosures, an even lightpeak NAS solutions to appear on the market. So if so, why buy now a usb 3.0 card when you can wait a year for a much faster and future proof solution?

So, your mac pro already has firewire 800 ports. You don't need to buy an additional card for them. Besides, it looks like you would like to have all your drives plugged in and avalailble all the time don't you? With the dozen of drives you"ll end up having its gonna be a pain to plug and unplug the ones you need and dont need at a any given time. You don't have that problem with fw 800 drives cuz they can be daisy chained, whilst usb 3.0 cannot. If you are not familiar with this concept it means that each drive can be connected to another drive, as of links of a chain, and only one end of the chain is actually connected to the mac pro. Of course the drives will still be bottle necked by the 800 mbps limit of the fw 800 port, but if I am not mistaken your mac pro has 4 fw 800 ports right? So If you end up with 8 external hard drives in fw 800 enclosures, you connect 2 of them in each fw 800 port and voila, your 8 drives are available to use ,anytime at the same time. Yes they wont be faster than usb 3.0 ones, but they will do fine until you get lightpeak ones. There are some fw 800 enclosures that come with RAID option ( at least in Japan there are) so each enclosure can hold two 3.5 hds and if they ae the same maker and model, they can be striped so they will be faster, maxxing out the 800 mbps transfer rate. If so, if you end up with 8 external hds, buy 4 of these enclosures and voila, you have them stacked and running at a pretty decent trnasfer rate until lightpeak comes.

Hope it helps.

 

[Ryan P]

Thanks. I Actually had forgotten about Firewire 800 as an option....although I do use it with my Macbook Pro....hmmm so what are the limits on daisy chaining? Other issue I could see is that when I have priced Firewire enclosures in the past they have always cost a fortune.

I was digging Lightpeak till I read about the whole copper bit, doesn't seem so uber cool anymore!

I guess in reality what I want is the lowest cost per GB solution, that I can also make into a reasonably tidy, quiet solution as I'm doing this all in my living room.

 

[EtherealMAC]

hmmm so what are the limits on daisy chaining?

You can daisy chain as many drives as you want, no limit on that end. The real limit is that regardless of how many drives are chained, they will always be limited by the ports's bottleneck, which means a 800 mbps transfer rate.

Other issue I could see is that when I have priced Firewire enclosures in the past they have always cost a fortune.

I dunno how is it in he US, check newegg and OWC Computing ones, the latter ones I've heard are pretty good.

I was digging Lightpeak till I read about the whole copper bit, doesn't seem so uber cool anymore!

By the copper issue you mean that initially, first generation lightpeak port may not feed power from the port? If that is your concern well, I guess you are worried about power usage of all the ac adapters of all your external drives, then? Well, if so you'll have the same problem with usb 2.0, 3.0 and even fw 800. All of these can feed power to an external drive, but the high perfomance 3.5 hard drive enclosures almost invariably need to be powered by an ac adapter anyways. Some usb ones that dont need ac adapters will still come with a Y cable for you to plug it into TWO usb ports, cuz they are so power hungry that one port wont do. I could be mistaken but the only fw 400/800 and usb 2.0/3.0 drives that dont need an ac adapter to work are low perfomance ones, specially the ones for low power 2.5 drives which it seems are not waht you are lookin for. So, If you are worried that first generation lightpeak might not bring copper ring and therefore cannot power your drives,hence needing ac adapters, well the same applies for usb 3.0. Thus, its a non issue.

I guess in reality what I want is the lowest cost per GB solution, that I can also make into a reasonably tidy, quiet solution as I'm doing this all in my living room.

The tidiest solution of all is a NAS box, but since those are pricey, I think daisy chaining fw 800 drives is way tidier than using usb hubs and having to plug/unplug usb drives everytime you want to mount/unmount them.

One more thing, by reading your original email it seemed like you were thinking on buying the lacie 3.0 hubs and the lacie 3.0 drives... you realize that if you do so you will also need to buy a usb 3.0 PCI card and install it in your mac pro right? Its not like the usb 3.0 hubs will automatically upgrade the transfer rate of your existing usb 2.0 ports. Therefore if you go the usb 3.0 route you'll need the PCI card+usb 3.0 hubs+usb 3.0 hd enclosures. That is way more messy than just daisy chaining a few fw 800s



Hope it helps

 

[Honumaui]

head over to www.avsforum.com and see what some of those people are doing for storage also ? might get you some other ideas

one other thought I am thinking of doing this with our house ? not sure yet the way to go

but limetech unRAID for storage has crossed my mind ?

I used to have a HTPC and just tied it into the network to access our office storage ? and thinking of doing that again so the noise stays in another room
also thinking about moving our storage to the closet and adding some fans to keep it cool and insulate it so less noise ?

 

[padapada]

DIY solution for lots of media: Unraid

 

[Honumaui]

funny I redid my post without refreshing great minds must think alike
hehehehe

the one other thing I have been debating for my HT setup is to rip very little ? store very little while I like having it on hand I have been thinking of using that money to just stream content and try that out a bit ?

rent bluray still but really I buy a few things that I love but really dont buy BR or DVD anymore now with 3D changing things and such the amount of money I would spend to store makes me wonder if its worth it or not ?

I do love hoarding my media and have thought rip in my maybe 400+ movies I have or just selling them off getting rid of them ?

I am really having a strugle with this the movies I really love streaming I might buy or rent a second time on BR and watch again ?

 

[onecajun]

Media storage

Build a storage server, use a free linux or nas operating system to run it. I would think cost might be a issue, but spending a little money would be better than stacking a bunch of external crap next to your computer. You would also offload all that crap from your primary system.

 

[Ryan P]

Thanks all good stuff, that Unraid is looking the most interesting of what I have seen so far. That is the lowest price per GB of anything I have seen. I need to do some more reading on it!

 

[ashman70]

How big are your movie files on average? What bitrate are they when you rip them? I have seen some amazing results on windows using a program called ripbot where an 8GB mkv file is reduced to 2-3GB with no visible loss of quality, I've seen the 2GB file played on a 58" plasma, action scenes show no loss of quality. I am not sure if Makemkv works the same on macs as ripbot, but it may be something worth looking into if your media files are huge.

 

[Ryan P]

How big are your movie files on average? What bitrate are they when you rip them? I have seen some amazing results on windows using a program called ripbot where an 8GB mkv file is reduced to 2-3GB with no visible loss of quality, I've seen the 2GB file played on a 58" plasma, action scenes show no loss of quality. I am not sure if Makemkv works the same on macs as ripbot, but it may be something worth looking into if your media files are huge.

I am, for the most part going, with full Bluray when possible. I haven't gotten much into the swing of recompressing for smaller but from downloading movies I can tell a pretty large difference between say a 8GB MKV on the net and a native Bluray. My screen is a 115" with a Panasonic AE4000 viewed from about 12' away, but I can also see the difference on my Cinema 27". I trim off the extra audio tracks so on average looking at around 25GB per movie I would say. So if you can fit 80 of those on a $80 dollar 2TB drive the storage cost is about a $1 per movie.

What really gets you is TV series. Mad Men is 115GB per season. I almost have Battlestar Galactica done and it is near 600GB's. Using MakeMKV for the converting which is quite speedy.

My general feeling is the storage cost will quickly come down and it will be worth the effort to do it at a high level right the first time. One day this will all fit on a damn thumb drive! Without real Bluray support in OS X it sort of forced me along this path, but now it has become something of a hobby, kinda cool to have someone over and have them be able to select from zillions of movies and shows with a click of a button!

From Lime Technology Website
"Note: all servers are currently sold-out. A new server product will be announced soon and available for shipping on Jan. 31, 2011."

I'd like to see what they announce, as building a server myself is rather difficult for me. I live on a Caribbean island. I finally managed to find a freight forwarder for newegg, but they tend to ship everything in different boxes from different warehouses that confuses it all. I had it cost me $40 shipping for a $20 USB key I added on as a extra last time. Ouch!

 

[onecajun]

Mr Picky

Yeah Ryan, I think you are picky just like me... I was going to say strip your file sizes down. Basically whatever you end up with will be costly due to your standards. have a good day like minded one!!!

 

[sarge-]

It wasn't quite my original intention, but my Mac Pro has turned into the center of a new home theater I've set up with Plex and a 1080p projector. I've been ripping bluray movies to disk, downloading from itunes, etc, etc and I am really struggling with disk issues. My photography and limited video work were nothing compared to this.

Right now I've got about 7 TB's of Media files and I don't see this addiction stopping anytime soon. I have a mixture of drives in my 2010 Hexcore that is maxed out and 1 drive literally hanging out the back on a SATA card.

You have similar concerns to mine. You might look at the thread I just started, but the basics are this:

Mac Pro 5,1 Hex 3.33 w/ a 15TB 5-bay eSATA NAS (5x 3TB drives). You could use DAS if you're not running other computers (I have a Mac Mini at each TV in the house).

My plan is to use the MP for 'work' attached to the NAS for media distribution (serves media even if the MP is off). ALL of this will then be backed up to a 30TB 10-bay DAS, also hanging off the MP.

So I'll have up to 12TB in the MP internally (+SSD boot and scratch) with a 15TB NAS, all backing up to a 30TB DAS. Something like $4,500 for the NAS and DAS.

Current thoughts/details and prices in this thread (click)

 

[nanofrog]

If your goal is primarily HTPC and a single system/HDTV of modest capacity (i.e. 8x or 10x bays), then you could use a Port Multiplier enclosure attached to the MP via eSATA (the Sans Digital TR8MP is an 8bay unit as an example; it even comes with the card). Using 3TB disks, you can get 24TB of raw capacity, and be able to use all of it via JBOD (done via Disk Utility - JBOD will be fast enough, as BR only needs 40MB/s for uncompressed data streams).

If it's to serve multiple systems/HDTV's, then you may want to look into a NAS (no need to leave the MP on either). The unRAID system by Limewire is a good way to go for a finished product, or you can build your own and buy their software.

Another DIY method (same concept Limewire is, but not use their software), is to build a system and run Linux or Open Solaris and run some form of ZFS file system for a large capacity, single storage pool (JBOD, or similar to parity but without the write hole issues, is Z-RAID 1 or 2 = comparable to RAID 5 and 6 respectively).

 

[Honumaui]

Thanks all good stuff, that Unraid is looking the most interesting of what I have seen so far. That is the lowest price per GB of anything I have seen. I need to do some more reading on it!

the avsforum also has some good info on people doing things with that ? so lots of good info out their on it

 

[Honumaui]

I'd like to see what they announce, as building a server myself is rather difficult for me. I live on a Caribbean island. I finally managed to find a freight forwarder for newegg, but they tend to ship everything in different boxes from different warehouses that confuses it all. I had it cost me $40 shipping for a $20 USB key I added on as a extra last time. Ouch!

what island you on ? you can PM me if you want

being from Maui used to the island stuff also lived on Utila for a year back in the 1990s (part of the Bay Islands of Honduras) taught scuba for 15 years so had lots of island friends one of my Maui buds now lives in the USVI wife and I have talked about moving down that way when our kids get a bit older ? but who knows I miss the South Pacific islands a lot