A question on Workflow and Hard-drives

[hvp]

Hi all,

Ok, here's the setup: I've just ordered a refurb MBP, which I will be putting an 256 SSD into. I have numerous externals, all 5400rpm afaik. I will be running Aperture as my primary cataloging and post-processing sofware.

The question is: What would be the best solution for where to store everything?

My initial thoughts are to import from the camera to the SSD and keep everything local while i'm working on the project, exporting the finals to an external and then moving the originals to an external when i'm finished with them, as this will/should maximise the speed benefit of the ssd while editing, but then not take up too much space when it's done. Does Aperture handle moving library source files from one disk to another nicely?

Any other thoughts on a good system for file management would be much appreciated.



Stu.

 

[Edge100]

Hi all,

Ok, here's the setup: I've just ordered a refurb MBP, which I will be putting an 256 SSD into. I have numerous externals, all 5400rpm afaik. I will be running Aperture as my primary cataloging and post-processing sofware.

The question is: What would be the best solution for where to store everything?

My initial thoughts are to import from the camera to the SSD and keep everything local while i'm working on the project, exporting the finals to an external and then moving the originals to an external when i'm finished with them, as this will/should maximise the speed benefit of the ssd while editing, but then not take up too much space when it's done. Does Aperture handle moving library source files from one disk to another nicely?

Any other thoughts on a good system for file management would be much appreciated.

Stu.

Can't comment on Aperture per se, but here's my file management system:

1. Import from camera to internal HD (actually, a 50GB partition on the fastest part of the drive) via FW800 card reader, into new Lightroom 3 catalog
1a. Create automatic backup of images on portable USB external, again via Lightroom. This ensures I have two copies of the RAW files.
1b. Erase memory card once import and backup are confirmed
2. Edit in Lightroom/PS
3. Export to JPEG. JPEGs are backed up on my SmugMug site and on Mozy.com
4. Import LR catalog and RAW files to master LR catalog on external 1TB drive (RAID 1, connected by eSATA).
5. Erase catalog and images on MBP drive

This system works well for me because I always have at least two copies of the RAW images, once they come off the memory card. It also allows me to use the (relatively) fast drive inside the MBP to actually work on the files (and you'd be even better off with an SSD), and then take advantage of RAID 1 redundancy to ensure that my files are safe once moved off the MBP. The LR catalog is also backed up, and the JPEGs go off-site.

The weak link here is that I don't have a true backup (RAID 1 is not a backup) of the master catalog and RAW files; if my home were to burn down or if the drive were stolen, I'd be sunk (though I'd have other concerns if my home burns down!). I'm thinking about adding an off-site backup via rotating USB drives or using a Voyager Q drive dock.

Hope this helps.

 

[flosseR]

working with referenced files.. ALWAYS That way you have the library on your fast disk and don't have to carry around the externals while still be able to work with your files...

Here are parts 1-3 of a guide i am writing. Part 2.5 covers referencing. This all keeps your library VERY small and saves your master files where you want.

If speed and proper organizing you want, that is the way to go.

 

[hvp]

working with referenced files.. ALWAYS That way you have the library on your fast disk and don't have to carry around the externals while still be able to work with your files...

Here are parts 1-3 of a guide i am writing. Part 2.5 covers referencing. This all keeps your library VERY small and saves your master files where you want.

If speed and proper organizing you want, that is the way to go.

Outstanding!

That's exactly the solution I was looking for.
Thanks so much for pointing me in the right direction.



Stu.

 

[flosseR]

No problem, sorry it is not 100% complete yet but I AM working on the missing parts .. writing is hard

 

[snberk103]

Also, head over to MacPerformanceGuide.com for loads of tips on optimizing PS and your Mac. In your case especially read his posts on SSDs, and how to keep them optimized. IIRC SSDs on Macs can easily lose speed over time if not kept "clean".

 

[a.jfred]

ALWAYS work on files straight from the hard drive, rather than an external drive - to do otherwise risks corrupting the external drive, thus destroying not just the photo you're working on, but all your saved photos too.

I actually have multiple back-ups: a 1TB drive (formatted for PC, from when I was running PC & Mac) that stores everything (music & photos), a 500GB portable drive for moving files to when I'm done working with them (or pulling them over to the HD to work on), and a separate drive that's stored off-site with everything, just in case my house burns down (sure, I could use something like dropbox, but for the same price, I get a safety deposit box, and can put a whole lot more in it).

 

[Edge100]

ALWAYS work on files straight from the hard drive, rather than an external drive - to do otherwise risks corrupting the external drive, thus destroying not just the photo you're working on, but all your saved photos too.

I have never, ever heard this before. Couldn't the same thing be said of the internal drive? Why would working on an external drive risk corrupting the drive? Isn't that what drives are for?

Whether you work internally or externally does not absolve you of the requirement for a proper backup regimen.

I actually have multiple back-ups: a 1TB drive (formatted for PC, from when I was running PC & Mac) that stores everything (music & photos), a 500GB portable drive for moving files to when I'm done working with them (or pulling them over to the HD to work on), and a separate drive that's stored off-site with everything, just in case my house burns down (sure, I could use something like dropbox, but for the same price, I get a safety deposit box, and can put a whole lot more in it).

This is a good strategy.

 

[snberk103]

ALWAYS work on files straight from the hard drive, rather than an external drive - to do otherwise risks corrupting the external drive, thus destroying not just the photo you're working on, but all your saved photos too.
...

I have never, ever heard this before. Couldn't the same thing be said of the internal drive? Why would working on an external drive risk corrupting the drive? Isn't that what drives are for?
....

I don't know what a.jfred was specifically thinking of.... but I can think of a couple ways. If the external drive was 'unmounted' during a write operation i.e. data or power cable comes loose.

Many people have their computer on an UPS or surge protector, but not their externals. If the power blips during a write operation you could see corruption in that case.

Or, many people never bother reformatting their externals from FAT.... which is easily corrupted. I'm sure there are others.

Most pro photographers I know are paranoid. Some don't delete their memory cards until they know that the images are on their working HD, plus 2 backups (i.e. they don't feel comfortable having less than 3 copies of their image files.) etc etc

 

[Edge100]

I don't know what a.jfred was specifically thinking of.... but I can think of a couple ways. If the external drive was 'unmounted' during a write operation i.e. data or power cable comes loose.

Many people have their computer on an UPS or surge protector, but not their externals. If the power blips during a write operation you could see corruption in that case.

Or, many people never bother reformatting their externals from FAT.... which is easily corrupted. I'm sure there are others.

Most pro photographers I know are paranoid. Some don't delete their memory cards until they know that the images are on their working HD, plus 2 backups (i.e. they don't feel comfortable having less than 3 copies of their image files.) etc etc

But wouldn't this preclude the use of externals altogether? They could come unplugged at anytime.

I did pro audio work for many years, and we always recorded to FW400 (and then FW800) externals, without issue.

I agree with working off the internal drive if you can, but there's absolutely no reason to avoid doing work from an external.

 

[snberk103]

But wouldn't this preclude the use of externals altogether? They could come unplugged at anytime.

I did pro audio work for many years, and we always recorded to FW400 (and then FW800) externals, without issue.

I agree with working off the internal drive if you can, but there's absolutely no reason to avoid doing work from an external.

Maybe photographers are just more paranoid???

 

[compuwar]

Or, many people never bother reformatting their externals from FAT.... which is easily corrupted. I'm sure there are others.

Having done a fair amount of computer forensics and data recovery, I'd have to say that (a) it's been a lot of years since I saw an OS corrupt a FAT table and (b) I'd rather have my data on a disk with a corrupted FAT than a corrupted NTFS, HFS+ or ext2-4 filesystem.

Paul

 

[flosseR]

ALWAYS work on files straight from the hard drive, rather than an external drive - to do otherwise risks corrupting the external drive, thus destroying not just the photo you're working on, but all your saved photos too.

I have no idea where you pulled this one from but from my own experience this is EXTREMELY far fetched. After close to 20 years as an IT person, this is the first time i read this. They are the same mechanical drives than inside your machine. The only possibility could be a break in the USB connection but then you MIGHT.. (read: very unlikely) get one file corrupted, not the whole disk...

Anyway, back on topic. with referencing you ARE working with files on your computer. In case you don't know. aperture (and Lightroom I think) creates a "working" copy of your RAW file and you work with that. Once you export it, Aperture will fetch the original RAW file, until then your referenced file is really untouched and could be located wherever you want.

You can even tell Aperture to back up your RAWs during import so you have a reference folder where all your masters are, a backup folder where the same masters get copied to and your Aperture library with which you work.
My 2c.

//f

 

[Edge100]

Maybe photographers are just more paranoid???

My experience is that they're about equal.

 

[Abraxsis]

ALWAYS work on files straight from the hard drive, rather than an external drive - to do otherwise risks corrupting the external drive, thus destroying not just the photo you're working on, but all your saved photos too.

I actually agree with this. Not only is it generally slower to work on files from an external drive I can PERSONALLY attest to the fact that a corruption can spread like wildfire. I worked on a triple redundant system prior to now. One primary backup (onsite), 2nd backup (onsite), and a 3rd backup (Offsite). 2nd and 3rd are rotated from my office to the bank vault once every two weeks. On the 1st of every month I bring them all together and sync them. Two weeks ago my main hard drive suffered complete failure right in the middle of the backup. Not only did the main drive crash, it corrupted the data on all three backups. Talk about tech related depression. Luckily, my brother and I were able to recover the data ourselves without a pricey recovery service. After some work I have devised my new plan ...
(I use the DNG format in case anyone was wondering)

Primary Storage: 1 TB main drive in the iMac (brand new)
Primary Backup: 1 TB RAID 1 device w/ 2x 1TB drives installed (onsite)
Secondary Backup: 500GB USB 2.0 External Drive (onsite)
Tertiary Backup: 500GB USB 2.0 External Drive (primary offsite)
Quaternary Backup: 500GB USB 2.0 External Drive (secondary offsite))

Rules:
1. Drives are never all in the same location for any reason.
2. Onsite Secondary backup is alternately swapped with either the tertiary and quaternary backups every 2 weeks.
3. All offsite drives travel out of the office in steel sided padded cases.

Is having 5 backup copies of my images overkill? Maybe ... but try to place yourself into the situation of believing you had lost 360GB of images, thousands of hours of PP, not to mention the travel and memories lost with personal shoots. Then the anal/paranoid setup sort of makes sense. You can build this setup yourself for less than 500 US. Of course bigger drives = more money naturally.

Tertiary and Secondary Backups are swapped