Saving Space With RAW Files

[haravikk]

Okay so I'm hoping to help someone I know who does photography mainly as a hobby, but to a professional level (he did work professionally, but on a much more expensive work-provided system). However, the main problem he's having is that storing all the RAW image files is taking up a lot of space on his older iMac with its 500gb internal drive. Now I know it's possible to upgrade the drive in an iMac, but it's not really something we'd want to do if we can avoid it, plus he isn't generating loads of images, we just want to squeeze more out of the current capacity really.

Anyway, I don't want to have to sacrifice the quality of the RAW files, but it occurs to me that there must be some other way to save them without the huge overhead it's having on the limited drive capacity (he's down to about 20gb free space!).

My first thought was simply taking RAW files and compressing them into a zip archive, but while this ought to give decent savings this isn't great for convenience as they'd no longer appear in searches, and it's extra effort to open them if they are needed.

I'm not super knowledgeable about other alternatives though; I understand that JPEG-2000 has some kind of lossless compression defined, but is this good enough to preserve a RAW file, or will other data be lost? Is it well-supported? Same question applies to PNG files, though I've also heard that these can actually end up larger than the original when dealing with such detailed files.

There's also Adobe's DNG, which seems like it would be a good thing to convert to regardless, purely for non-device specific nature of them. But do these support lossless compression?

He's got some 400gb of files, the biggest portion of which is RAW images, so even modest compression savings should make a big difference in the short term at least.

 

[chrfr]

My first thought was simply taking RAW files and compressing them into a zip archive, but while this ought to give decent savings

Raw files are already compressed. With my Canon 5D Mark II raw files, zip saves less than a megabyte. Not worth it.
The reality is that raw files take up a lot of space. It's a necessary tradeoff, and the only real solution is more storage. A couple of external drives would do fine.

 

[haravikk]

Raw files are already compressed. With my Canon 5D Mark II raw files, zip saves less than a megabyte.

That must be vendor-specific then, as I just took a few sample files, and zipped 100mb of RAW files down to 78mb, which is a decent saving.

I'm not sure what exact model of camera he's using, but the files have been accumulated for a while so there are probably a bunch of different RAW formats in the collection.

The biggest problem is that he's not the most tech-savvy, so while external drives are an option, I'm not sure that organisationally it'd be the best idea, particularly when it comes to handling back-ups etc. For someone who can do tons of intricate editing in Photoshop, give him one error message in Mail and I'm called in, constantly

I'm hoping to find something I can eventually script and just leave to run periodically, reducing sizes even modestly. While I know about various image formats, I'm not really big on the pros and cons regarding photography.

 

[Fujiko7]

+1 to external drives. They are cheap enough now.

I just bought a couple of these. I know you say your friend isn't tech savvy, but these are just plug and play via USB. Buy a couple and use one for backups.

 

[chrfr]

I'm hoping to find something I can eventually script and just leave to run periodically, reducing sizes even modestly. While I know about various image formats, I'm not really big on the pros and cons regarding photography.

External storage or a larger internal disk really is the only appropriate option. Raw files are "sacred." Only DNG is even potentially an alternate image format, which may or may not reduce the file size. Any other format so greatly reduces the options on what can be done with the images that they should all be disregarded. The exception to this is that if the photographer is only using the Raw images as components of composited or heavily retouched images that the original image is no longer relevant, perhaps then the raw files could be deleted or converted to a new format.

 

[dwig]

That must be vendor-specific then, as I just took a few sample files, and zipped 100mb of RAW files down to 78mb, which is a decent saving. ...

Correct.

Some cameras save compressed RAW files and some don't. Many offer a the user a choice.

Compressed RAW, as done in camera, is a lossy compression method but the loss is very very small and does not induce any of the sharpening artifacts common with JPEG compression. If you're having space problems and your camera offers the option then opting for compressed RAW is a good choice. Of course, this will not help with the existing RAW files, only with new images.

Perhaps the best approach is an external hard drive and a DAM, like Lightroom, that can work with "offline" images. You can put older images on the external drive and only those is regular current use on the internal. LR5 can, if configured correctly, maintain previews for all of the images so that they can be worked with, other than export, when the external HD is offline (e.g. you're in the field showing work to a client and selecting images for a project, ...).

 

[haravikk]

External storage or a larger internal disk really is the only appropriate option.

It still seems a bit like throwing hardware at the problem; even if I can free up 70-80gb it'd probably be more than enough until the next computer upgrade, plus it requires organisation and I suspect more free USB ports than his iMac has (it's got two printers and scanners, plus a Time Machine drive already hanging off of them ), and it's an older iMac so only USB2.

Raw files are "sacred." Only DNG is even potentially an alternate image format, which may or may not reduce the file size.

I grabbed myself a small set of some of the images I'm looking at compressing, and DNG actually seems like it'll be the way to go as it appears to be doing some form of lossless compression as standard as the files are coming out around half their original size.. It also seems to be better supported by OS X, as the Finder is generating high-detail Quicklook previews (not just the thumbnails), and Preview can open them without complaint.

I'm going to be wiping the system for a clean install of Mavericks anyway (finally get him mostly up-to-date with a view to doing Yosemite once some of the lingering issues are fixed). I might see if I can use the Adobe DNG Convertor to convert the files straight from back-up while I'm restoring them as it should save a big chunk of space.

If it's not enough I'll recommend some new external drives to give more capacity.


On a related note, are there any good options for TIFF and PSD files? I know TIFF supports LZW compression and it seems like it should be pretty well supported, just tried a couple of 75mb photograph scans and they shrank down to under 20mb with LZW enabled. It's not noticeably slower to save, and actually seems faster to open (less to read from the disk I expect).

 

[OreoCookie]

It still seems a bit like throwing hardware at the problem; even if I can free up 70-80gb it'd probably be more than enough until the next computer upgrade, plus it requires organisation and I suspect more free USB ports than his iMac has (it's got two printers and scanners, plus a Time Machine drive already hanging off of them ), and it's an older iMac so only USB2.

In this case that's because your friend really needs the hardware: RAW files take up a lot of space, and OS X doesn't take kindly to having only 20 GB free. And it is very easy to generate
Stuff like zipping RAW files is not a solution, especially if your friend uses one of the more popular DAM software out there (e. g. Lightroom), because zipping files interferes with how they work. The cheapest stop gap measure is to get an external harddrive with all the caveats that you list, but the question is what the alternatives are: even if you clean up your friend's computer so that he has ~100 GB of space, that doesn't leave much room. It's quite easy for me to take 8~10 GB worth of photos in a single day. I'll leave the math to you, it's just a matter of weeks to months until your friend feels the pressure from dwindling storage space again.

What are the other alternatives to an external hard drive?
(1) Your friend could get a NAS (e. g. a 2-bay Synology system such as the DS214 filled with two 4 TB hard drives).
(2) Your friend could have the internal hard drive replaced with one that has plenty of space to spare (I recommend a 4 TB drive).
(3) Your friend could get a new Mac with more storage space (in that case I would suggest getting a 3 TB fusion drive). But that's about it.

Since your friend runs into slow downs with regular tasks, IMO he needs a new machine.

 

[edjs]

I'm not super knowledgeable about other alternatives though; I understand that JPEG-2000 has some kind of lossless compression defined, but is this good enough to preserve a RAW file, or will other data be lost? Is it well-supported? Same question applies to PNG files, though I've also heard that these can actually end up larger than the original when dealing with such detailed files.

With my Canon raw files and Lightroom, exporting an image with maximum jpeg quality (but still lossy) results in a file that is roughly 50% the size of the original raw. Converting to dng or zipping the raw file results in negligible drop in file size, so the original raw is already compressed.

Since you've already found the raw files you are dealing can be compressed further, you may find even better savings than I saw in converting to jpeg. However, even with lossless jpeg you will lose information. There is a lot of sensor data in the raw that allows one to make adjustments to the image easily; any editing choices will be 'baked-in' when the raw image is converted.

 

[The Bad Guy]

Just get your buddy to delete all the less-than-stellar images he's hoarding. Problem solved.

If he's anything like me or just about every other person I've met, there's at least five times as many photos in his library as there should be.

Still going to need an external drive for backups though. That's important.

 

[simonsi]

Move your current backup drive to a new (say) 2TB external, move his photo files/library to the existing backup drive (if it isn't 1TB then replace it too).

When you upgrade his machine simply move the drives to the new setup, nothing lost.

Yes it is throwing hardware at it, that is because he needs new hardware but drives are a lot cheaper than a new machine.

Compressing a mix of RAW file formats will likely take a substantial amount of time/effort and possibly cause issues when (if!) he needs to access them.

 

[bgd]

(1) Your friend could get a NAS (e. g. a 2-bay Synology system such as the DS214 filled with two 4 TB hard drives).

This was my approach and has worked out well for me. I no longer factor computers into my storage requirements. My MBA was struggling due to lack of space and now has a new lease on life. External backup drives hang off the NAS so everything is much tidier.

This approach may give more life to his existing machine, it has for mine.

 

[OreoCookie]

This was my approach and has worked out well for me. I no longer factor computers into my storage requirements. My MBA was struggling due to lack of space and now has a new lease on life. External backup drives hang off the NAS so everything is much tidier.

This approach may give more life to his existing machine, it has for mine.

I'll also get one in the near future. I've tried a Transporter, but the software was slow, buggy and in the end it wasn't flexible enough. I'll probably get either a Synology DS214 or a DS214+ and fill it with one disk at first. This way, my gf and I have access to some shared data (e. g. movies), and I can access everything from all over the world. And like you say, it's a solution that's independent of the computer you use.

Edit: I just ordered a DS214+ and a 4 TB HGST NAS hard drive. I'll keep you guys updated on how it goes.

 

[OreoCookie]

Still going to need an external drive for backups though. That's important.

Good that you mention that, this has escaped my mind.

 

[monokakata]

Does your friend's iMac have a FW800 port? FW800 drives/enclosures are somewhat more expensive than USB ones, but if he has a port and he's not using it -- that's a good solution.

I too have a friend who's running out of space on a 500 GB iMac, and is hoping for a magic solution. He doesn't want an external, but has no good reason for not wanting one. He just doesn't. And thus he has no solution except a new internal, which he also doesn't want to do.

People do get themselves into these binds, don't they?

 

[bgd]

Good that you mention that, this has escaped my mind.

I have 2 4TB drives in the NAS, the second acts as a mirror. I also have an external drive (eSata) which backs up the important stuff - this is handled by the NAS. So that is 3 onsite copies and to round off I use CrashPlan.

Basically it's set up and forget - for now

 

[OreoCookie]

I have 2 4TB drives in the NAS, the second acts as a mirror. I also have an external drive (eSata) which backs up the important stuff - this is handled by the NAS. So that is 3 onsite copies and to round off I use CrashPlan.

Basically it's set up and forget - for now

Does Crashplan run on the NAS itself?

 

[bgd]

Does Crashplan run on the NAS itself?

No. I run that from my computer.

 

[haravikk]

No. I run that from my computer.

I actually managed to get CrashPlan running on my Synology NAS; it's a bit annoying to setup though and officially unsupported, but it lets me run the online backups overnight when the various machines in my household are asleep or shutdown, rather than having to make sure they can wake when needed and have four or more machines fighting over bandwidth.


Okay, so regarding the RAW files, I was doing a clean install on the system to upgrade to Yosemite, so when restoring the files I used Adobe DNG Convertor to move across all raw files (converting them) and separately synced everything else with rsync (excluding the original RAW files). It's saved around 60gb, so the originals definitely seem to have had no compression in the first place, and there don't appear to be any issues with quality.

There's still a ton of uncompressed PSD and TIFF files; I'm not sure what I can do with the PSDs, but re-saving TIFFs with LZW compression enabled makes a big difference for most files I've tested, so I'm going to see if I can use a script to do it automatically (ImageMagick supports this I believe), but I'll have to test that to make sure it's not inviting disaster.


Anyway, I appreciate all the responses, so many thanks for these; though I am still going to finish trying to compress stuff before I consider recommending new hardware as it doesn't worth switching to a terabyte if the space is still being wasted on overly large files. There also seems to be advantages to the compression, as the DNG files open much quicker, with no loss of detail, and TIFFs with LZW seem to have similar results; less time waiting for the hard-drive I suppose, the cost of decompression is less overall.

 

[OreoCookie]

I actually managed to get CrashPlan running on my Synology NAS; it's a bit annoying to setup though and officially unsupported, but it lets me run the online backups overnight when the various machines in my household are asleep or shutdown, rather than having to make sure they can wake when needed and have four or more machines fighting over bandwidth.

I've read instructions online, but it's not quite clear to me what shares are being backed up. From what I understand you need to keep running Crashplan on at least one client, is that correct?

 

[haravikk]

I've read instructions online, but it's not quite clear to me what shares are being backed up. From what I understand you need to keep running Crashplan on at least one client, is that correct?

Not exactly; basically CrashPlan has two components, the backup daemon, and the UI. Normally you run both of these on the same machine, but when you run a "headless" system, the backup daemon runs on that (your NAS) all the time, and you just tell the UI how to connect to it so you can view progress, change settings etc., so you might have the app on one of your machines for managing stuff, but the full process isn't needed.

The tricky bit is deciding how to get your data onto the NAS system; if you use Time Machine, then it will actually use a disk image, so CrashPlan backs that up (it can't see individual files), whereas CrashPlan running on your machines will backup individual files. I'm using rsync to get around this, but that's got its own issues.

Easiest way to do it is probably to setup headless CrashPlan on the NAS and configure it to act as a backup target, then back-up everything shared with it. Then on your systems you can install CrashPlan and use the NAS as a target. To do this you'll need an extra copy of the CrashPlan app, since you'll configure that one to connect to the NAS for management. Still a bit confusing I know, unfortunately it's something that still isn't exactly easy

 

[OreoCookie]

Not exactly; basically CrashPlan has two components, the backup daemon, and the UI. Normally you run both of these on the same machine, but when you run a "headless" system, the backup daemon runs on that (your NAS) all the time, and you just tell the UI how to connect to it so you can view progress, change settings etc., so you might have the app on one of your machines for managing stuff, but the full process isn't needed.

[...]

Easiest way to do it is probably to setup headless CrashPlan on the NAS and configure it to act as a backup target, then back-up everything shared with it. Then on your systems you can install CrashPlan and use the NAS as a target. To do this you'll need an extra copy of the CrashPlan app, since you'll configure that one to connect to the NAS for management. Still a bit confusing I know, unfortunately it's something that still isn't exactly easy

If I understand you correctly, this setup still only backs up the volumes that have been mounted to the client (which you use to configure the background process), correct? This configuration seems semi-officially supported, but it is a bit of a hack.