Photographer/Film Maker here - extremely confused about route to go for storage.

[matthewh133]

Hi everyone, I do photography for a living and thus have terabytes upon terabytes of 42 megapixel photos and 4k video files (Sony A7R III, Canon 1DX II). Currently, I have a very inefficient method of storing these files on external hard drives and just having to plug in whatever hard drive contains the files that I need at that time.

What I want to do is have all of my files stored on one device that is permanently attached to my computer so I can access them on a whim. Also, I want this device to be fast so that I am not having issues with Lightroom, Photoshop, Premier Pro etc lagging because of the connection type.

I'm thinking that right now I want a device that can hold up to around 48tb+ (24tb for storage and 24tb for backup/copy).

What would you suggest for my needs? Just an external USB 3 hard drive enclosure with 4 x 12tb hard drives? Would this be slow to edit on? I have a separate SSD in my PC that I can copy Premiere Pro projects and their corresponding video files onto to edit when working on a particular project and then move them off of afterwards, but I'm guessing this hard drive enclosure solution should be more than enough for photo editing off of and storage?

Any thoughts would be greatly appreciated as I'm a little lost right now. I should mentioned I'm also hoping to spend around $1500 (maximum $2000) if possible.

 

[jackapple]

First, what computer do you have, so you can take advantage of the best connection type (Thunderbolt 2, 3, just USB 3)?

I'm in the same boat deciding on storage options for my future iMac Pro, and I've narrowed it down to the following TB3 setups:

• LaCie 6big
• Promise Pegasus3
• OWC ThunderBay 4 x Soft RAID - I currently have 2 of these (archive + offsite backup) + 2 ThunderBay minis, each with 4 x 1TB SSDs (RAID 0 stripe for working data)

Check out Mac Performance Guide, which is an enormous resource in helping you optimize your workflow and backup.

 

[matthewh133]

First, what computer do you have, so you can take advantage of the best connection type (Thunderbolt 2, 3, just USB 3)?

I'm in the same boat deciding on storage options for my future iMac Pro, and I've narrowed it down to the following TB3 setups:

• LaCie 6big
• Promise Pegasus3
• OWC ThunderBay 4 x Soft RAID - I currently have 2 of these (archive + offsite backup) + 2 ThunderBay minis, each with 4 x 1TB SSDs (RAID 0 stripe for working data)

Check out Mac Performance Guide, which is an enormous resource in helping you optimize your workflow and backup.

This is actually for my PC (I own a 2016 Macbook Pro I use on the road but I'm looking for a solution for my desktop right now). I only have USB 3 unfortunately. I've read it shouldn't make a huge difference for my purposes though if I'm using the SSD in my PC for editing 4k video files and just using the hard drive enclosure for storing and editing the photo files?

I'm interested in why the need for an expensive RAID system like you've linked vs a basic hard drive enclosure? Obviously you have automatic backing up of files on the RAID, but why not save the money and have a $150 HDD enclosure and 4 x 12tb hard drives (2 for main storage and 2 duplicates for backup, then obviously a corresponding drive off-site)?

 

[kohlson]

Indeed, why do people spend all that money on RAID? Mostly, because it works better than the alternative. Generally, RAID fits a price/performance/workflow/reliability-safety goal. For example, you could go all SSD, and this would be plenty fast and reliable. These can max out a USB3 port, but are too expensive once you start looking at multiple TBs. HDDs have good price/GB, but a single consumer-grade HDDs don't have the performance or reliability. Download (free) Black Magic Disk Speed Test, and run it on your MBP. It will probably check all the boxes. Now look at a disk speed utility for your PC. Not even close. The only way to get HDDs to acceptable IO performance is to use multiple (RAID) at a time. As well, some RAID implementations can provide a measure of safety against drive issues.

12TB drives are $500-ish, so I'm not sure how this will fit a $1500 budget. You'll need some sort of 4- or 6-bay enclosure. You could certainly use a 2TB SSD working drive, and copy files back and forth - assuming the wait time isn't too long.

I'm not trying to be negative here. If you can make a fast SSD/slow HDD combo work, there's nothing wrong with that. But people spend money for RAID on demanding storage requirements like 4K because it's faster/more reliable than the alternative for their workflow.

 

[matthewh133]

Indeed, why do people spend all that money on RAID? Mostly, because it works better than the alternative. Generally, RAID fits a price/performance/workflow/reliability-safety goal. For example, you could go all SSD, and this would be plenty fast and reliable. These can max out a USB3 port, but are too expensive once you start looking at multiple TBs. HDDs have good price/GB, but a single consumer-grade HDDs don't have the performance or reliability. Download (free) Black Magic Disk Speed Test, and run it on your MBP. It will probably check all the boxes. Now look at a disk speed utility for your PC. Not even close. The only way to get HDDs to acceptable IO performance is to use multiple (RAID) at a time. As well, some RAID implementations can provide a measure of safety against drive issues.

12TB drives are $500-ish, so I'm not sure how this will fit a $1500 budget. You'll need some sort of 4- or 6-bay enclosure. You could certainly use a 2TB SSD working drive, and copy files back and forth - assuming the wait time isn't too long.

I'm not trying to be negative here. If you can make a fast SSD/slow HDD combo work, there's nothing wrong with that. But people spend money for RAID on demanding storage requirements like 4K because it's faster/more reliable than the alternative for their workflow.

Thanks for the post, I definitely appreciate your input. Regarding price, I can get 4 x 12tb Barracuda Pro hard drives for $899, and then a HDD enclosure compatible with eSATA and USB 3 for $150, totaling $1050. I would probably add a 1tb Samsung EVO 850 HDD (connected inside the PC VIA SATA and used for 4k editing) costing about $330, bringing the total to just under $1400.

What I do know is that photo files seem to load fast and allow me to edit no problems with my current setup with them just on external USB hard drives or on the WD 3.5" SATA drives I have in the PC... so I figure I just need a device that will give me at least the same speeds, but which will allow me to have them all inside one enclosure and plugged in with a single cable. 4K video on the other hand is new to me, but I'm assuming that as long as I have a separate SSD for the 4k video and project files while editing in Premiere Pro, I shouldn't really be seeing any major negatives right? I'm normally pretty good with computers, having built many and worked on them since I was 6 years old, but all of this storage stuff just seems to go over my head.

Someone told me that I didn't need a RAID for my needs and I wouldn't see any benefit if I was backing up the files on 2 separate drives (one backup would be in the enclosure, one off-site). They said that RAID was mainly used by business' to keep networks and systems running with minimal downtime in case of HDD failure. I do understand that RAID 0 on the other hand multiples the speeds of the hard drives, but the cost of these systems for the amount of storage I need is around $5000 without even including the price of the drives... AND from what I'm reading on Amazon none of them are even reviewed that favorably. I'm interested in your thoughts on this if you have the time.

Regarding the following "The only way to get HDDs to acceptable IO performance is to use multiple (RAID) at a time. As well, some RAID implementations can provide a measure of safety against drive issues." are you referring to RAID 0? Where there's no redundancy but it pulls the hard drives together to multiple the speeds?

 

[xgman]

If you go with normal high capacity drives in nay config, I suggest you only consider Enterprise rated drives. HGST seems to consistently have the highest reliability ratings for what it's worth.

anyway a good resource for reviews and stuff is http://www.storagereview.com/

 

[jackapple]

This is actually for my PC (I own a 2016 Macbook Pro I use on the road but I'm looking for a solution for my desktop right now). I only have USB 3 unfortunately. I've read it shouldn't make a huge difference for my purposes though if I'm using the SSD in my PC for editing 4k video files and just using the hard drive enclosure for storing and editing the photo files?

I'm interested in why the need for an expensive RAID system like you've linked vs a basic hard drive enclosure? Obviously you have automatic backing up of files on the RAID, but why not save the money and have a $150 HDD enclosure and 4 x 12tb hard drives (2 for main storage and 2 duplicates for backup, then obviously a corresponding drive off-site)?

I followed Lloyd Chamber's advice the last few years with having a Master and Archive workflow—fast SSD Master RAID volume + Archive HDD Volume, with at least 2 local cloned backups of each (Carbon Copy Cloner clones). Here's what it all looks like:

Late 2014 iMac w/ 1TB SSD

• 1TB Boot volume - system data + primary work

ThunderBay mini (4 x 1TB OWC 6G Enterprise SSDs in a RAID 0 stripe - Thunderbolt 2)

• 4TB Master volume - current photo/video/audio projects

ThunderBay 4 (4 x 8TB HGST Ultrastar He8, individually partitioned - Thunderbolt 2)

1. 8TB HGST - 6TB Archive / 1TB Boot Clone / 1TB Time Machine
2. 8TB HGST - 6TB Archive Clone / 1TB Boot Clone 2 / 1TB Cache (used for macOS High Sierra caching)
3. 8TB HGST - 4TB Master Clone / 4TB Archive Clone 2
4. 8TB HGST - 4TB Master Clone 2 / 4TB Extra (temporary copying/transferring/etc without risk)

At an offsite location, I have an old Mac mini Server:

• 120GB OWC 6G Enterprise SSD - Boot
• 2TB 6G Electra SSD - Server (data)

ThunderBay 4 attached via ThunderBolt 2 acting as a backup via Resilio Sync:

1. 5TB Toshiba - 5TB Archive Backup
2. 5TB Toshiba - 4TB Master Backup / 1TB Cache
3. 5TB Toshiba - 2.5TB Time Machine / 2TB Server Clone / 500GB Boot Clone
4. 5TB Toshiba - 500GB Boot Clone 2 / 2TB Server Clone 2 / 2.5TB Extra

I have another ThunderBay mini (4 x 1TB Samsung 850 EVO SSDs) that I used as a second RAID Master volume for a bit before changing to this workflow. I thought I was going to use it for my wife when my iMac goes to her, but I'm likely going to be selling both ThunderBolt mini SSD setups + SoftRAID when I get the iMac Pro.

 

[matthewh133]

I followed Lloyd Chamber's advice the last few years with having a Master and Archive workflow—fast SSD Master RAID volume + Archive HDD Volume, with at least 2 local cloned backups of each (Carbon Copy Cloner clones). Here's what it all looks like:

Late 2014 iMac w/ 1TB SSD

• 1TB Boot volume - system data + primary work

ThunderBay mini (4 x 1TB OWC 6G Enterprise SSDs in a RAID 0 stripe - Thunderbolt 2)

• 4TB Master volume - current photo/video/audio projects

ThunderBay 4 (4 x 8TB HGST Ultrastar He8, individually partitioned - Thunderbolt 2)

1. 8TB HGST - 6TB Archive / 1TB Boot Clone / 1TB Time Machine
2. 8TB HGST - 6TB Archive Clone / 1TB Boot Clone 2 / 1TB Cache (used for macOS High Sierra caching)
3. 8TB HGST - 4TB Master Clone / 4TB Archive Clone 2
4. 8TB HGST - 4TB Master Clone 2 / 4TB Extra (temporary copying/transferring/etc without risk)

At an offsite location, I have an old Mac mini Server:

• 120GB OWC 6G Enterprise SSD - Boot
• 2TB 6G Electra SSD - Server (data)

ThunderBay 4 attached via ThunderBolt 2 acting as a backup via Resilio Sync:

1. 5TB Toshiba - 5TB Archive Backup
2. 5TB Toshiba - 4TB Master Backup / 1TB Cache
3. 5TB Toshiba - 2.5TB Time Machine / 2TB Server Clone / 500GB Boot Clone
4. 5TB Toshiba - 500GB Boot Clone 2 / 2TB Server Clone 2 / 2.5TB Extra

I have another ThunderBay mini (4 x 1TB Samsung 850 EVO SSDs) that I used as a second RAID Master volume for a bit before changing to this workflow. I thought I was going to use it for my wife when my iMac goes to her, but I'm likely going to be selling both ThunderBolt mini SSD setups + SoftRAID when I get the iMac Pro.

All of that looks like it's going to be WELL over what I'm hoping to spend.

 

[jackapple]

All of that looks like it's going to be WELL over what I'm hoping to spend.

I was just providing insight into my personal setup since you asked about the purpose behind RAID vs. a standard JBOD enclosure. Hopefully it gives you another way of looking at your workflow and backup strategy.

This can be accomplished in any number of ways, but the ultimate goal is to have a fast working drive (which you already do with your SSD), an archive volume (speed is mostly irrelevant), and then a combination of local and offsite backups so you're protected in the event of a catastrophic loss. Your data is priceless. We'd all be wise to treat it accordingly.

 

[kohlson]

are you referring to RAID 0? Where there's no redundancy but it pulls the hard drives together to multiple the speeds?

For HDDs, the only way to get fast I/O (in this case, speed) is to have some form of Raid 0. SSDs can saturate traditional storage buses (USB3, SATA-3), HDDs cannot without RAID. (Note that Apple began moving away from SATA many years ago - too slow.) Adding drives together increases performance, and decreases reliability. Various other forms of RAID (such as 1+0, 5, 6) balance performance and reliability. All this demands an expensive controller card (host bus adapter) to keep everything working. You're right - it adds up quickly.

A friend's cMP had an SSD boot, plus 5 internal drives, and a 4-bay external which he runs as JBOD through eSata. Plus a dock. His workflow is similar - he copies his sources and libs to various drives, and makes backups. This works for him. But he spends a lot of time waiting for copies to complete. And he works only in HD.

I don't see any reason why it wouldn't work for you.

 

[gr2020]

You didn't say what your computer was - but if if you have (or can have) 10Gb ethernet, such as with an iMac Pro, then I would personally consider something like a Synology 1817+ (an 8-bay NAS, ~$900 + drives) with a 10GbE network interface card (~$200?), and a small 10GbE switch (~ $200). Depending on how you configure the NAS, this setup should be capable of around 1 GB/sec of transfer, and since it's on your network, you can use it from any of your other devices.

 

[matthewh133]

You didn't say what your computer was - but if if you have (or can have) 10Gb ethernet, such as with an iMac Pro, then I would personally consider something like a Synology 1817+ (an 8-bay NAS, ~$900 + drives) with a 10GbE network interface card (~$200?), and a small 10GbE switch (~ $200). Depending on how you configure the NAS, this setup should be capable of around 1 GB/sec of transfer, and since it's on your network, you can use it from any of your other devices.

I have a PC, I'm in the process of buying a new motherboard that supports thunderbolt 3. It seems all of this is kind of irrelevant right now as the best my PC supports is eSATA and USB3, none of which are all that fast and most RAID systems capabilities would be wasted.

I'm now looking at the https://www.bhphotovideo.com/c/prod...ology_p3r8hd48wus_pegasus3_r8_48tb_8_bay.html and https://eshop.macsales.com/item/OWC/TB2IVT48.0S/ and thinking I may just have to bite the bullet and fork out the money. This should last me another 5+ years of storage I would think, so it might just be worth the extra dollars. Does anyone have opinions on these two? They seems to both be Thudnerbolt 3 compatible and 48tb, why is the Pegasus3 almost twice the price?

 

[gr2020]

I have a PC, I'm in the process of buying a new motherboard that supports thunderbolt 3. It seems all of this is kind of irrelevant right now as the best my PC supports is eSATA and USB3, none of which are all that fast and most RAID systems capabilities would be wasted.

I would think you could get a 10GbE NIC for your PC in the form of a PCI Express card, and you'd be good to go with the setup I suggested, if you wanted to go that way.

 

[matthewh133]

I would think you could get a 10GbE NIC for your PC in the form of a PCI Express card, and you'd be good to go with the setup I suggested, if you wanted to go that way.

Good to know.

So essentially the four options are:

(NAS) Synology DS1817+ with 4x12tb hard drives (also 4 spare HDD slots for expansion) + Ethernet 10gb swtich and PCIE card - $3300

OR

(RAID) OWC Thudnerbay 4 48tb (includes 4x12tb hard drives) - $2999

OR

(RAID) Promise Technology Pegasus3 R8 PC Edition 48TB 8-Bay Thunderbolt 3 (includes 48tb of hard drives) - $5499

OR

(RAID) Lacie Thunderbolt 4 48tb (6 x 8tb hard drives) - $3699

Does anyone have any thoughts on any reason to go with one over the other? Assuming my PC can handle Thunderbolt 3 for the RAIDs or the ethernet cable for the NAS?

 

[jackapple]

I have a PC, I'm in the process of buying a new motherboard that supports thunderbolt 3. It seems all of this is kind of irrelevant right now as the best my PC supports is eSATA and USB3, none of which are all that fast and most RAID systems capabilities would be wasted.

I'm now looking at the https://www.bhphotovideo.com/c/prod...ology_p3r8hd48wus_pegasus3_r8_48tb_8_bay.html and https://eshop.macsales.com/item/OWC/TB2IVT48.0S/ and thinking I may just have to bite the bullet and fork out the money. This should last me another 5+ years of storage I would think, so it might just be worth the extra dollars. Does anyone have opinions on these two? They seems to both be Thudnerbolt 3 compatible and 48tb, why is the Pegasus3 almost twice the price?

I can't say enough good things about the ThunderBay and OWC products in general. Here's a video review by Max Yuryev.
[doublepost=1517532233][/doublepost]

Good to know.

(RAID) OWC Thudnerbay 4 48tb (includes 4x12tb hard drives) - $2999

Tip: If you don't care about OWC "burning in" the drives for you and instead certifying them yourself, you can save a lot by purchasing the enclosure and drives separate from B&H (especially on taxes if you don't live in NY).

Seagate 12TB Enterprise HDD x 4 @ $439.50 = $1,758

ThunderBay 4 enclosure = $449.99

Total = $2,207.99

 

[matthewh133]

I can't say enough good things about the ThunderBay and OWC products in general. Here's a video review by Max Yuryev.
[doublepost=1517532233][/doublepost]

Tip: If you don't care about OWC "burning in" the drives for you and instead certifying them yourself, you can save a lot by purchasing the enclosure and drives separate from B&H (especially on taxes if you don't live in NY).

Seagate 12TB Enterprise HDD x 4 @ $439.50 = $1,758

ThunderBay 4 enclosure = $449.99

Total = $2,207.99

Oh wow, good to know! That's quite the saving. Thanks a lot. May I ask what "burning in" the drives entails? Is this something I need? Something that's easy to do myself? Also, how do those enterprise drives compare to say the Barracuda Pro drives? I heard the Barracuda Pro 12tb drives are meant to be some of the fastest out there.

 

[OBirder]

While I spent the money on the iMac Pro, I use a very simplified method for my external HDD's.

I use a 10 port USB 3 hub. I operate currently 9 external HDD's on it with capacity between 4 and 6 TB each. I buy my drives from Costco for $100 each (when on sale) and no question asked warranty for 1 year.

I only use currently 8 TB for photos and make dual backup. For the backup I use Syncovery which copies file. For PC I used Bvckup which is very simple and efficient. The beauty is that it only has to copy newer files and not full backups. It runs twice a day in the background and is basically not noticeable. On one backup I keep deleted files for 90 days, while the second backup is an exact mirror. The advantage if a drive fails I can just point the photos to the other drive and I am immediately running.

I have a drive failure once or twice a year. One third of the failures is under warranty. For the others I just pay my $100 to replace it and since I have dual backup it's not a problem. In addition I am getting a new drive with current capacity for the drive.

In addition I use external 4 TB portables for a 3rd and 4th backup (mirror) and they are weekly alternated for offsite storage.

I keep my catalog on an external USB C Samsung T5 SSD which provides the speed. I have a second low cost SSD on which I keep current projects I am working on. This way the speed of the external HDD's doesn't really matter for me. It's long term storage and backup. And even if I need a photo from the long term storage I have direct access in LR and it is really not slow at all.

Another advantage: If I go to an event, I take my 2 external mirror drives, connect them to my Macbook and can operate from there without putting my actually drives at exposure or risk. Even if I accidentally change something during the event, once I am back home, within minutes my software makes it an exact mirror of the main drives.

Disadvantage: Cable clutter - However I run a 9 ft USB 3 extension cable and have a solution where the cables and drives are hidden, but still have airflow.

Cost of my method:
- 8 x 6 TB drives = 48 TB = $800
- Hub = $50
Total: $850

Since you have already drives this might be a consideration for you.

 

[Michael Scrip]

What would you suggest for my needs? Just an external USB 3 hard drive enclosure with 4 x 12tb hard drives? Would this be slow to edit on? I have a separate SSD in my PC that I can copy Premiere Pro projects and their corresponding video files onto to edit when working on a particular project and then move them off of afterwards, but I'm guessing this hard drive enclosure solution should be more than enough for photo editing off of and storage?

As others have said... it basically comes down to two types of data:

• Current Projects
• Archives

You want your current working projects to be as fast as they can be... so you work from a fast SSD or MVNe drive.

Then when you're finished... you move them to slower, and larger, (and cheaper) mass storage. And then you make duplicates/backups of that.

But if you you want high-speed and huge storage... it can get VERY expensive. And that's probably not a road you want go down. It might be unnecessary.

Personally... I'm in the market for a Synology NAS for my archive data. Like you... I've got quite a few hard drives piling up with all my old photo and video projects (I use Premiere Pro and Lightroom). So I want to consolidate it all to a single device to hold everything. A Synology NAS will do that.

It won't be particularly fast... but that's not its job. It's for archive purposes.

My current projects would still be edited on fast internal SSDs... but then put away for long-term storage.

I can still access older files on the slower drives (or future NAS). If I need to go back to an old project for some heavy re-editing... I'd move it back to my fast current projects drive temporarily. Though, for me, it's rare that I need to revisit a complete project from a few years ago to do any major editing. At most... I might only grab a couple video clips or a few photos to do a quick edit.

In other words... I don't need my archives to be fast. Just big.

Question... how much data do you generate per year?

My biggest year was less than 2TB of data. And my current total archive is around 6TB of video files and Canon RAW files. I'm small potatoes.

My plan is to start with a 4-bay Synology NAS and two 8TB drives. If I use Synology Hybrid RAID... that will give me 8TB of redundant storage. And then I'll also get an 8TB external drive for a backup of the NAS.

The nice thing about a Synology NAS and its Hybrid RAID is that you can pop in another drive and quickly expand its storage with minimal downtime. As I approach its 8TB limit... I can pop in another 8TB drive and double my storage to 16TB. (and then I'd add another 8TB external drive to mirror the newly-added data)

Then I'll still have one empty bay for the future... allowing me a total of 24TB in the one Synology box. That will keep me going for a while at my current rate!

But it sounds like you're starting with 24TB of data... so you're already jumping into the deep end of the data pool. That's not gonna be cheap, as you know.

I still believe in the concept of fast working drives and slower/cheaper archive drives, though. I don't see the need to keep older infrequently-accessed files on expensive fast storage.

Yeah... you can spend $5,000 on a huge 8-bay super-fast RAID box to keep everything you've ever done ready at high-speed. But how many of those old files will sit there basically untouched for years?

I've got files from 15 years ago... but they don't need to occupy space on fast expensive drives. That's the way I look at it anyway.

I guess you've gotta ask yourself: how much space do I need... and do I need every file to be fast?

You can get the 48TB 8-bay Thunderbolt 3 box and have all 48TB available at high-speed for $5,500

Or you can get a slower NAS box with the same 48TB of capacity for $2,600. (of course you edit your current and recent projects on faster working drives)

The whole reason you want so much space is to hold everything. But do you need your entire archive always available at high-speed?

I'll come home with 50GB of RAW photos or 200GB of video. Those get edited on my fast working drive. But after the projects are delivered... they get moved off to cold storage. And I'll probably never see them again.

I can still get to those old files... but I'm not paying high rent for all those old files.

 

[Fishrrman]

OP wrote:
"Regarding price, I can get 4 x 12tb Barracuda Pro hard drives for $899, and then a HDD enclosure compatible with eSATA and USB 3 for $150, totaling $1050. I would probably add a 1tb Samsung EVO 850 HDD (connected inside the PC VIA SATA and used for 4k editing) costing about $330, bringing the total to just under $1400."

Just wondering, but...
... once you have this, what are you going to use to back it up?

Seems to me that whatever you decide to buy... you're going to have to buy TWO of them.
One for primary storage, and the second for backup?

 

[matthewh133]

I guess you've gotta ask yourself: how much space do I need... and do I need every file to be fast?

You can get the 48TB 8-bay Thunderbolt 3 box and have all 48TB available at high-speed for $5,500

Thanks a lot for taking the time to write this post, it's very much appreciated. I can't remember if I mentioned it before, but multiple times a week generally I have to access and edit older photos from the archive. This is the main reason I'd like the speeds of my archive to still be pretty fast. I'm not sure if you've seen the option below, but you can get a thunderbolt 3 RAID from OWC with 48tb of hard drives for only $2200 (suggested to me by another user).

https://eshop.macsales.com/item/OWC/TB3IVKIT0GB/

+

https://www.newegg.com/Product/Prod...seagate_enterprise-_-1Z4-002P-006N4-_-Product

= $2200 for 48tb of RAID storage.
[doublepost=1517595437][/doublepost]

While I spent the money on the iMac Pro, I use a very simplified method for my external HDD's.

I use a 10 port USB 3 hub. I operate currently 9 external HDD's on it with capacity between 4 and 6 TB each. I buy my drives from Costco for $100 each (when on sale) and no question asked warranty for 1 year.

I only use currently 8 TB for photos and make dual backup. For the backup I use Syncovery which copies file. For PC I used Bvckup which is very simple and efficient. The beauty is that it only has to copy newer files and not full backups. It runs twice a day in the background and is basically not noticeable. On one backup I keep deleted files for 90 days, while the second backup is an exact mirror. The advantage if a drive fails I can just point the photos to the other drive and I am immediately running.

I have a drive failure once or twice a year. One third of the failures is under warranty. For the others I just pay my $100 to replace it and since I have dual backup it's not a problem. In addition I am getting a new drive with current capacity for the drive.

In addition I use external 4 TB portables for a 3rd and 4th backup (mirror) and they are weekly alternated for offsite storage.

I keep my catalog on an external USB C Samsung T5 SSD which provides the speed. I have a second low cost SSD on which I keep current projects I am working on. This way the speed of the external HDD's doesn't really matter for me. It's long term storage and backup. And even if I need a photo from the long term storage I have direct access in LR and it is really not slow at all.

Another advantage: If I go to an event, I take my 2 external mirror drives, connect them to my Macbook and can operate from there without putting my actually drives at exposure or risk. Even if I accidentally change something during the event, once I am back home, within minutes my software makes it an exact mirror of the main drives.

Disadvantage: Cable clutter - However I run a 9 ft USB 3 extension cable and have a solution where the cables and drives are hidden, but still have airflow.

Cost of my method:
- 8 x 6 TB drives = 48 TB = $800
- Hub = $50
Total: $850

Since you have already drives this might be a consideration for you.

Interesting, thanks for this. I guess it is possible, it's not really slow to edit in Lightroom from external drives. I would probably prefer to spend a couple hundred extra to put all of the hard drives in an enclosure and have one cable instead of 8 though haha. I would still have to buy the hard drives, I need my current externals for traveling storage needs as I can be gone long periods at a time. Maybe this could work for photo/video archive, and I can get a separate RAID 0 with SSDs for 4k video editing?
[doublepost=1517595562][/doublepost]

OP wrote:
"Regarding price, I can get 4 x 12tb Barracuda Pro hard drives for $899, and then a HDD enclosure compatible with eSATA and USB 3 for $150, totaling $1050. I would probably add a 1tb Samsung EVO 850 HDD (connected inside the PC VIA SATA and used for 4k editing) costing about $330, bringing the total to just under $1400."

Just wondering, but...
... once you have this, what are you going to use to back it up?

Seems to me that whatever you decide to buy... you're going to have to buy TWO of them.
One for primary storage, and the second for backup?

For every drive that I buy, I'll be buying a corresponding drive as a backup.

 

[matthewh133]

Just as an update, I ended up deciding on the OWC ThunderBay4, with 4 12tb Seagate Enterprise drives. This is only about $2200 which isn't too bad.

 

[joema2]

...I'm thinking that right now I want a device that can hold up to around 48tb+ (24tb for storage and 24tb for backup/copy)....

I have three OWC Thunderbay 4 32TB RAID-0 arrays, a primary, a backup and one is off site. They work fine. For Windows I'm less familiar with how USB 3 would handle multiple arrays of that size. The data rates to each array peak at about 600 MB/sec. I guess if you had multiple USB controllers, each feeding a separate USB array that might work.

The problem with putting 4 x 12TB drives in a single chassis is the fear of some common failure. If the chassis goes down everything is down. With a primary and backup chassis you can cut over to the backup.

From a capacity standpoint you ideally need at least 20% free space for spinning drives. So if your actual media storage need is 24TB you'd physically need closer to 32TB to house that.

 

[h9826790]

IMO, a good long term solution is a multi bay NAS. Pick one that has USB 3.0 and 10Gbps network support. Also, pick a model that support SSD cache.

e.g. You need 24TB, get a 5 bay NAS, 4x8TB in RAID 5 provide 24TB actual usable storage with reasonable redundancy. The 5th bay insert a SSD to provide high IOPS cache.

10Gbps connect provide ~1000MB/s transfer speed (USB 3.0 is about 500MB/s). Even though your PC don't have 10Gbps connection now, it's very easy to add a 10Gbps PCIe network card. If you don't want that now, USB 3.0 still provide decent performance for just storage or photo work. And you can still upgrade to 10Gbps in future.

Then get another identical NAS as backup. If you want to lower the cost, a cheaper 4bay NAS that only has 1Gbps port should able to provide proper backup. Again, 4x8TB HDD in RAID 5.

But if budget available, it's better to get another identical NAS as backup, not for performance. But in case the primary NAS failed, you can simply switch to the backup NAS and continue to work with the same performance.

 

[Sirmausalot]

I just solved that same setup. I got 4X10Tb drives as that was the best price performance. The extra 2TB per drive was too dear. Make sure you get 7200 RPM drives with 256MB cache each. This is what I got from NewEgg: Seagate IronWolf 10TB NAS Hard Drive 7200 RPM 256MB Cache SATA 6.0Gb/s 3.5" Internal Hard Drive ST10000VN000

Get 2 OWC RAID USB 3 enclosures (either from OWC directly or B&H) and make sure your format them as RAID 0 for performance. But one will be your video editing drive and the other will be backup. The advantage to the backup being raid stripped is if the first drive goes down, you can immediately resume work. The whole setup cost me around $1400

you do not need thunderbolt for spinning disks. Only for stripped SSDS.

Doug

Hi everyone, I do photography for a living and thus have terabytes upon terabytes of 42 megapixel photos and 4k video files (Sony A7R III, Canon 1DX II). Currently, I have a very inefficient method of storing these files on external hard drives and just having to plug in whatever hard drive contains the files that I need at that time.

What I want to do is have all of my files stored on one device that is permanently attached to my computer so I can access them on a whim. Also, I want this device to be fast so that I am not having issues with Lightroom, Photoshop, Premier Pro etc lagging because of the connection type.

I'm thinking that right now I want a device that can hold up to around 48tb+ (24tb for storage and 24tb for backup/copy).

What would you suggest for my needs? Just an external USB 3 hard drive enclosure with 4 x 12tb hard drives? Would this be slow to edit on? I have a separate SSD in my PC that I can copy Premiere Pro projects and their corresponding video files onto to edit when working on a particular project and then move them off of afterwards, but I'm guessing this hard drive enclosure solution should be more than enough for photo editing off of and storage?

Any thoughts would be greatly appreciated as I'm a little lost right now. I should mentioned I'm also hoping to spend around $1500 (maximum $2000) if possible.

 

[pixelatedscraps]

Just another photographer chiming in their two cents of how I work -

All six computers in my studio (1x iMP, 2x cMP from 2010, 3x MBP) run the same Dropbox Plus (1Tb) + extended version history (up to 1 year) where we store Lightroom Catalogs, Final Client Images, Portfolios (keynote and PDF), Inspiration / Moodboards and Client Proposals, etc. Once everything is synced, any user can edit and pick up the catalog wherever they are in the world - provided they have a hard drive with the raw files. All raw images (we use 2x Sony A7R III and a Canon 1D X II) are stored on external hard drives and on the server.

Our Lightroom catalogs are stored on our Dropbox Plus (1TB with 12 month backup) which is synced across all six computers, meaning any one of us can work on catalogs at different times (or cities). We keep all raw files on external SSD or HDD (see below) with any final imagery delivered to the clients exported to Dropbox for delivery. It's not an ideal system (and neither is Adobe Lightroom) but it has worked well for us. We are currently using Capture One for tethered shoots with the raw imagery then being imported to Lightroom (which we're just better / more adept at).

In terms of storage, we utilise the following items - with the Extreme 900 SSD and Chronosync being the new additions to our system.

Day-to-day / on shoot storage for this year:

• 1x SanDisk Extreme 900 1.92TB SSD 'Master' drive. This is where all our current work is imported to, processed and then exported from.
• 2x 5TB LaCie Rugged TB2 / USB-C backup hard drives. These are backups of the Extreme 900 Master drive above, backed up using Chronosync once a week. My partner (art director) and I (managing director) each have one of these drives while the Master either stays connected to the iMac Pro for editing / export or comes on shoots with us.
• 2x 5TB Seagate Backup Plus (backups of the LaCie Ruggeds above). One of these stays at home while the other stays in a safety deposit box and is updated monthly (we'll see about that!).

Long term storage:

• Synology DS416Play (4x 8Tb WD Red in SHR RAID setup for 24TB) - this new NAS contains everything from 2017 onwards.
• Synology DS414j (4x 8Tb WD Red in SHR RAID setup for 24TB) - this is an older NAS containing everything from 2014-2016.
• Orico 4-bay USB3.0 SATA HDD enclosure - random assortment of 3.5" and 2.5" HDDs from before 2014. Looking at around 20Tb in total scattered across the globe.
• No cloud backup solution as of now, no bank vaults either but I just purchased two small bright yellow Pelican cases which may either go into a safety deposit box at the bank or...be stuffed under the bed

For next year, I want to upgrade our internal network to a 10Gb one. Thus looking at the following:

• 8-bay Synology DS2015xs Diskstation (10Gb, SSD caching, will most likely start off with 4x 12TB Ironwolf drives in SHR Raid to begin with for budget reasons)
• Netgear 12-port 10Gb switch