Size Matters?

[DucatiTerminator]

First a confession: I'm probably the worst when it comes to organizing photos, and I have quite a few of them spread over several drives, computers, and optical (and other media) of various types. I can generally find whatever I'm looking for, but lately it's been taking a bit longer, and I've finally decided to consolidate my media and start the path towards better organization.

At a quick glance, I have a couple of TB of images at arms reach, and I'm guessing I have double to triple that if I were to consolidate all of my images (and double+ that again if I factor in digital video). Recently, I started shooting more seriously again, and I'm starting to add images at a fairly fast rate. I shoot everything RAW, and until now, the file sizes haven't been too large, however, I am thinking of adding a Canon 5Ds(R) to the arsenal and am concerned how fast the size of my library will start growing. Maybe some of you D800/D810 users could chime in? God (and wife) willing LOL, I'm also hoping to add the new Phase One XF camera which will most likely reset my perception of storage.

I'm curious what you hardcore enthusiasts/prosumers/pros do for storage and back up. Any of you have working libraries over a couple of terabytes, and if so, how do you manage your media? Any thoughts, advice, or recommendations would be greatly appreciated!

 

[The Bad Guy]

Do what I did and delete the crap you're never going to look at again. That'll free up a at least half of your storage.

 

[Apple fanboy]

Deleting stuff you don't need is a good strategy, and one I don't use enough.
One of the reasons I chose the D750 over the D810 was file size. 24 MP is plenty for anything I'm going to print.
Yes storage is relatively cheap these days, but the more drives you have, the longer it takes to find stuff!

 

[tomnavratil]

I also shoot mostly RAW so there's an increasing need of storage space. Deleting old stuff is definitely a good idea however if you think you might ever need it, convert all those old RAW files into JPGs. Also my Nikon D610 shoots at maximum of 24MP so it's not too bad storage-wise and as Apple fanboy mentioned - alright for almost any print.

When it comes to storage itself, I use a simply external HDD, backed up through Time Machine to another one however with your needs that might not be sufficient for sure. You might want to look into more sophisticated and professional solutions such as DAS - for example some from Drobo - http://www.drobo.com/storage-products/5d/ or LaCie - https://www.lacie.com/products/product.htm?id=10607 - where you can swap disks to larger sizes if you need to.

 

[ChrisA]

Do what I did and delete the crap you're never going to look at again. That'll free up a at least half of your storage.

Yes. Culling your library is also a good way to improve the overall quality. It's easy to delete maybe 2 out of every 10. But most people can cut further. Removing the lest good image makes the entire collection better.

Next, you need some kind of software to manage your library. I'd recommend Adobe Lightroom as the best choice for use on a Mac right now. Just put everything in the LR library and start entering media data as you have time

As for how to store that much data. Get a RAID box of some kind.
Synology makes some good ones
https://www.synology.com/

If you know a little about computers you can build a top-tier storage server inside a PC server chassis using this software
http://www.freenas.org/for-home/

But very importantly -- You need to BACK UP your data. RAID is not a backup. Ant system that follows ALL of these rules will work:
1) The data must exist on at least THREE different physical media at all times (A raid system counts as one physical media)
2) The data must exist at a minimum of TWO different geographical locations at all times.
The above needs to apply especially during a backup copy operation. So if you keep an off site copy (rules #2) you need three copy to rotate.

I use
1) the primary storage on the iMac (physical media #1)
2) Time Machine connected to the iMac (physical media #2)
3) Some disconnected hard drives I keep in a fire safe in the same house (physical media #3)
4) An on-line backup service (Backblaze for $5 per month) (geographic location #2)

 

[ChrisA]

...
Yes storage is relatively cheap these days, but the more drives you have, the longer it takes to find stuff!

This is why you need some kind of library manager. I use Apple's Aperture but it is now discontinued and I plan to move to Adobe Lightroom. Using something like either of these it does not matter how many photos you have. As long as you took the time to enter basic media data you can search. Aperture and others can even search for faces.

I tried and experiment by looking through my collection for images of George Washington and Aperture found some shots I have of a $1 bill and of Mount Rushmore. If you camera has GPS or if you take time to tag your files your can search on location. Some software now can do even better and find objects (other then faces) in images.

 

[tomnavratil]

But very importantly -- You need to BACK UP your data. RAID is not a backup. Ant system that follows ALL of these rules will work:
1) The data must exist on at least THREE different physical media at all times (A raid system counts as one physical media)
2) The data must exist at a minimum of TWO different geographical locations at all times.
The above needs to apply especially during a backup copy operation. So if you keep an off site copy (rules #2) you need three copy to rotate.

I use
1) the primary storage on the iMac (physical media #1)
2) Time Machine connected to the iMac (physical media #2)
3) Some disconnected hard drives I keep in a fire safe in the same house (physical media #3)
4) An on-line backup service (Backblaze for $5 per month) (geographic location #2)

This.

People don't realise how important backup is until they lose their data. There is also a lot of people who think that RAID is a backup even when it's not.

 

[ChrisA]

When it comes to storage itself, I use a simply external HDD, backed up through Time Machine to another one however with your needs that might not be sufficient for sure.

The big problem with your backup system is that is is not protecting you from the most common causes of data loss

1) Theft of the equipment (the #1 cause) because your Time Machine disk is right there with the rest of the equipment
2) A power hit, say lightening hits a power line within a mile of your house. Anything plunged in the AC mains is going to get fried.
3) natural disaster or a house fire. (EVERYONE who had a fire said "It probably will never happen to me.")
4) Software bug that corrupts a file or an entire folder or drive. (This does happen)
5) Operator error. (You do something dumb.)

 

[tomnavratil]

The big problem with your backup system is that is is not protecting you from the most common causes of data loss

1) Theft of the equipment (the #1 cause) because your Time Machine disk is right there with the rest of the equipment
2) A power hit, say lightening hits a power line within a mile of your house. Anything plunged in the AC mains is going to get fried.
3) natural disaster or a house fire. (EVERYONE who had a fire said "It probably will never happen to me.")
4) Software bug that corrupts a file or an entire folder or drive. (This does happen)
5) Operator error. (You do something dumb.)

Oh no, I wasn't describing my full back-up but what I simply use for images as I don't need a RAID or DAS/NAS due to the size of my library.

I still back-up my most important files to the cloud service and also files off-site.

I agree with your points for sure. Having all files in one place is not a great idea. However even having Time Machine enabled as a basic backup is something that many users don't even do, which really surprises me these days. So yes, a multiple backup, including an off-site one is a great idea but I know that most users don't even do the basics.

 

[dwig]

All good backup strategies include some form of off-site backup, period. If off-site backup is not included then it is not a good strategy.

 

[DucatiTerminator]

Unfortunately, deletion isn't the best option for me. For example, one of my dear friends is a model who is not very computer savvy. The content of her alone is approaching a terabyte, and I do almost all of her post-processing. I'm a father of three active boys, and I'm guessing family content is approaching 2 terabytes (or more); my better half would be slightly upset if I even considered deleting any of it, lol. And I do have some commercial content which I can't delete.

Even if I were to pare down the library, it won't address my current/future requirement for storage. I've had great experiences with LaCie in the past; I will look into both Drobo and Synology. I was hoping to delay the investment in pro level storage, but I think the writing is on the wall.

 

[DucatiTerminator]

Next, you need some kind of software to manage your library. I'd recommend Adobe Lightroom as the best choice for use on a Mac right now. Just put everything in the LR library and start entering media data as you have time

As for how to store that much data. Get a RAID box of some kind.
Synology makes some good ones
https://www.synology.com/

If you know a little about computers you can build a top-tier storage server inside a PC server chassis using this software
http://www.freenas.org/for-home/

But very importantly -- You need to BACK UP your data. RAID is not a backup. Ant system that follows ALL of these rules will work:
1) The data must exist on at least THREE different physical media at all times (A raid system counts as one physical media)
2) The data must exist at a minimum of TWO different geographical locations at all times.
The above needs to apply especially during a backup copy operation. So if you keep an off site copy (rules #2) you need three copy to rotate.

I use
1) the primary storage on the iMac (physical media #1)
2) Time Machine connected to the iMac (physical media #2)
3) Some disconnected hard drives I keep in a fire safe in the same house (physical media #3)
4) An on-line backup service (Backblaze for $5 per month) (geographic location #2)

I've very recently acquired Lightroom through Adobe CC. Many, many years ago, I casually used Extensis Portfolio, but never really adopted it like I should have. Upon initial use, I noted that I have 10K+ images I imported into the catalog. I'm guessing I will be well into the six figures by the time all is said and done.

I have some local drives set up in a couple of RAIDs, but it seems like a larger box like you and Tomas mention is inevitable. As far as back up is concerned, I've used Time Machine on a couple of the Macs, but I haven't yet attempted to use it for the larger drives with everything mounted to the computer. I have used older tape drives, but VXA2 is pretty old and slow and the cleaning tapes are getting difficult to source without spending a mint. I could probably make use of a LTO-5 drive I have and keep the media in a fire safe. I will look into Backblaze; the unlimited data sounds nice. I'm concerned that because of a rather slow office internet connection (dsl availability only at the moment) that backing up that much data could take a really, really long time.

 

[steveash]

My system isn't very high-tech but it works for and has done for many years. I am a commercial photographer and a graphic designer before that so have plenty of client files to look after. I also us a Hasselblad with its rather large raw files.

My approach is to use an ordered system of multiple Lightroom catalogs. I have one per client and for personal work create a new one each year. I keep my file system in order with a folder for each catalog. I can then archive off old work onto external drives (2 copies, 1 taken off site) once the project is done. Everything is catalogued with Disk Catalog Maker so I always know where my files are.

I aim to produce quality over quantity and so my libraries are never huge. I do still go through each shoot and delete the mistakes, duplicates and bad shots. For what is left I have never needed more than 1TB of storage for current work and so have never bothered with an expensive and potentially catastrophic (if some of the stories I've heard are correct) RAID system. I don't need instant access to work produced years ago.

 

[DucatiTerminator]

My system isn't very high-tech but it works for and has done for many years. I am a commercial photographer and a graphic designer before that so have plenty of client files to look after. I also us a Hasselblad with its rather large raw files.

My approach is to use an ordered system of multiple Lightroom catalogs. I have one per client and for personal work create a new one each year. I keep my file system in order with a folder for each catalog. I can then archive off old work onto external drives (2 copies, 1 taken off site) once the project is done. Everything is catalogued with Disk Catalog Maker so I always know where my files are.

I aim to produce quality over quantity and so my libraries are never huge. I do still go through each shoot and delete the mistakes, duplicates and bad shots. For what is left I have never needed more than 1TB of storage for current work and so have never bothered with an expensive and potentially catastrophic (if some of the stories I've heard are correct) RAID system. I don't need instant access to work produced years ago.

Thank you (and everyone else!) for your input. I like your approach; I think I can use a modified version of it as a part of my workflow. I do a little bit of commercial work, and this will likely work very well.

I don't think I will every be able to limit the quantity aspect as much as I would like. Between shooting a friend's band and my kids' sports alone (my volunteer position for years has been club photographer), I tend to fill up space pretty quickly, even after trying to be as selective as possible regarding the "keepers," and I'm only shooting at 12-16MP at the moment. As I mentioned earlier, I'm thinking of adding Canon's 50MP body (and/or hopefully a Phase One XF -- my dream) to the arsenal, so either way, my storage needs will grow significantly. Archiving old work makes sense, but I do like to keep a lot of the fun/hobbyist (majority) readily accessible as I am asked to access it by friends and family fairly often.

Love your work, BTW!

 

[Robotti]

I started using Amazon Cloud Drive for backup as soon as they published it, and have been satisfied so far. It's unlimited and a good price for that. It also has very good upload speeds (at least from where I am), which seta it apart from many other services I've tried. Their interface for accessing and cataloging the files for online access needs some work, but I think it'll get there. For backup purposes, more than sufficient already.

 

[OreoCookie]

I'm not quite sure I understand why you think you need a 35+ MP camera, because that's a really expensive proposition. A camera like the Canon 5D R needs the best glass money can buy, and a storage system to match. These high MP RAW files are BIG, and eat up a lot of space. You will have to look into getting a RAID for sure. I recommend Synology here, they have the best software and excellent hardware. I would recommend their 5- or 8-bay systems (1515+ and 1815+ or 2015xs, respectively), and populate these with good NAS hard drives (e. g. those of HGST). However, this will just get you raw storage, and you need to up your game with organization. Perhaps it's better to just accept that your past life is a mess, start with a clean sheet, and apply all the best-practices to new photos.

 

[Apple fanboy]

I use LR to catalogue all my images. However I'm not as commited to key wording etc as I should be!

 

[DucatiTerminator]

I'm not quite sure I understand why you think you need a 35+ MP camera, because that's a really expensive proposition. A camera like the Canon 5D R needs the best glass money can buy, and a storage system to match. These high MP RAW files are BIG, and eat up a lot of space. You will have to look into getting a RAID for sure. I recommend Synology here, they have the best software and excellent hardware. I would recommend their 5- or 8-bay systems (1515+ and 1815+ or 2015xs, respectively), and populate these with good NAS hard drives (e. g. those of HGST). However, this will just get you raw storage, and you need to up your game with organization. Perhaps it's better to just accept that your past life is a mess, start with a clean sheet, and apply all the best-practices to new photos.

Believe me, I don't NEED a 35+ MP camera. I simply WANT a new full frame camera that can also shoot DX should I like, shoots and focuses more quickly and accurately, and gives me the best IQ I can afford. I shoot with a 5D which meets my requirements right now, and have ready access to MKII and MKIII bodies, but I've had a lot of fun shooting with a Fuji X-Pro1 and XT-1 lately; IQ, glass quality, and portability are on par or exceed my current DSLR gear in most of my situations, and file sizes are relatively small. I do suffer from GAS sometimes, but I rarely indulge in gear on a whim. The 5D is getting tired, and it's speed can be limiting at times, so I figured it's time for an upgrade. I know I will have to upgrade some of my glass, but will likely rent or borrow a lot in the beginning.

I'm using 4TB HGST (2 million hour MTBF) drives at the moment in my aging cMP. I will definitely look into Synology, thanks. My personal life is organized chaos at best, lol, but I have really come a long ways in improving the "organized" part. My professional life is actually a lot more organized than it appears.

BTW, my mother is from Nagasaki. I haven't been to Fukuoka in many, many years, but really loved it there. I'm sure it's changed a lot since.

 

[DucatiTerminator]

I use LR to catalogue all my images. However I'm not as commited to key wording etc as I should be!

This is one of the things I'm trying to be very disciplined about as a LR newbie.

 

[Apple fanboy]

This is one of the things I'm trying to be very disciplined about as a LR newbie.

Good idea. Too late for me 3 years in!

 

[Melizard]

I keep it simple: One 1 TB harddrive as my primary storage for photos only (I have only ~450 GB of photos), and one 2 TB harddrive for backup (with other data). One harddrive lives at home, the other at the office. I don't take photos regularly (just when I travel), so whenever I have a new photo folder I back it up immediately. I don't keep anything on my computer and I keep every photo I take. It's not a perfect system, but I'm not a professional photographer and I feel secure enough with this system.

As far as archiving goes, I just keep simple folders with descriptive names (organized by year). I have subfolders that separate out the crap photos from the good ones. It's pretty easy for me to find things, but then again I've only been shooting for 5 years and all of my photos represent events.

 

[maflynn]

Do what I did and delete the crap you're never going to look at again. That'll free up a at least half of your storage.

I need to do that more, but with storage being so cheap, I've had the mentality of just a bigger drive. I'll not go back and remove my old photos, but I think I'm starting to be a bit more aggressive in my culling process.

 

[OreoCookie]

Believe me, I don't NEED a 35+ MP camera. I simply WANT a new full frame camera that can also shoot DX should I like, shoots and focuses more quickly and accurately, and gives me the best IQ I can afford.

Personally, I think this is a perfectly legitimate reason, I just wanted to say that the costs you incur go beyond that of the body. Personally, I love the new Fuji cameras, I have a X100s and it's a camera I fell in love with. In a perfect world I'd probably get an X-Pro 2 next. In any case, I hardly use my dslr anymore (I have a full assortment of lenses, 2 flashes, etc.). The only problem I have with it is that my wife loves the X100s, too, and I hardly get to use it anymore

I'm using 4TB HGST (2 million hour MTBF) drives at the moment in my aging cMP. I will definitely look into Synology, thanks.

I also use HGST NAS drives in my Synology NAS (4 TB and 6 TB), according to Backblaze's stats, they are the most reliable.

 

[MCAsan]

I open my cards with Perfect Photo Suite Browser. It lets me quickly cull by seeing the jpg previews in the raw files. Then I drag and drop the keepers onto Lightroom for import. The remaining shoots are lost when I reformat the card in the camera. The Lightroom import preset fills out my copyright data and invokes a Develop preset that does all kinds of basic adjustments I do to all images (add clarity, sharpen, S curve..etc.) The preset also generates the 1:1 previews. I do not convert native raw to DNG format. The import process does not take long because there are much fewer images imported than images on the card. Once the import is done it is quick to do final edits as the 1:1 previews have already been created and to allow quick zooming.

 

[Miltz]

First a confession: I'm probably the worst when it comes to organizing photos, and I have quite a few of them spread over several drives, computers, and optical (and other media) of various types. I can generally find whatever I'm looking for, but lately it's been taking a bit longer, and I've finally decided to consolidate my media and start the path towards better organization.

At a quick glance, I have a couple of TB of images at arms reach, and I'm guessing I have double to triple that if I were to consolidate all of my images (and double+ that again if I factor in digital video). Recently, I started shooting more seriously again, and I'm starting to add images at a fairly fast rate. I shoot everything RAW, and until now, the file sizes haven't been too large, however, I am thinking of adding a Canon 5Ds(R) to the arsenal and am concerned how fast the size of my library will start growing. Maybe some of you D800/D810 users could chime in? God (and wife) willing LOL, I'm also hoping to add the new Phase One XF camera which will most likely reset my perception of storage.

I'm curious what you hardcore enthusiasts/prosumers/pros do for storage and back up. Any of you have working libraries over a couple of terabytes, and if so, how do you manage your media? Any thoughts, advice, or recommendations would be greatly appreciated!

Reading this makes me laugh a bit. My pictures all are all the place. But it's a organized mess. I can never not find something. So when I got my first Mac to use as a pro (I've had macs before but not for my photography) I was in a complete shock when the OS doesn't give me the same access to my photos like in Windows 7 Pro. I like to see my stuff at the file level and organize it that way. So long story short Mac OSX didn't work for me on a pro level, I just loved Windows "messy" file system. Storage wise I shoot RAW as well and always add internal storage to my PC which is not so easy on the newer MACS. And I also have a external 2TB which is an additional back-up. Brb... I need to go do a manual back-up right now. (Yes dragging my files across drives )And do what works for you.