My Photo collection is about 400 GB and have over 15,000 images. I have what I think is a great backup system. My system is Files live on a iMac and are backed up weekly (or with major changes) to a mac mini server running crash plan, the mini server is also backed up locally daily. I also have a off site backup at my work and a disk I carry around with me in my bag. All backup are refreshed every week. So I think I am covered. However I have been reading about data rot on hard drives and it got me thinking, some of my images I may not open let alone re-save for years. How can I protect my collection from data rot. The backup are made with either super duper or rsync. And OS X has not built in data checking so if a file has gone bad the bad copy just migrates through the backups. Or is Data rot something that is so rare that it is not worth worrying about.
Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Photo archiving - long term storage and data rot
- Thread starter dimme
- Start date
- Sort by reaction score
its very real on optical disks. you don't know how long lasting your cd-rs and dvd-rs are until you wait. i saw special archival quality optical disks at frys once
dont trust ssd drives, they can self erase when the power is out. if you read the ssd drive warranties , they say we only guarantee data retention of 6 months. there was a study where someone bought a bunch of ssd drives and stored themvwith no power , and every single one of them was erased in 18 months
if your pictures are that important, then perhaps you should get a network attached storage that has RAID . that way you can swap out hard disks.
if you want true archival lifetime , then you might have to print them. thats what the Smithsonian recommends. heh
maybe you should think about a commercial cloud company.
usb enclosures are the worst. unless you installed a driver, the usb enclosure blocks s.m.a.r.t. so while your drive may be failing, your operating system has no idea, because the drive isn't attached directly to its sata port.
i heard a lot of hard disks failing, and getting bad sectors, and eventually crashing. i never heard of magnetic / mechanical hard disk rot of data
yeah. if your photos are that important , i think you should just go ahead and pay apple the 9.99 a month for 1 terrabyte of iCloud until iCloud gets discontinued.
i don't think even iCloud will be around in 25 years
dont trust ssd drives, they can self erase when the power is out. if you read the ssd drive warranties , they say we only guarantee data retention of 6 months. there was a study where someone bought a bunch of ssd drives and stored themvwith no power , and every single one of them was erased in 18 months
if your pictures are that important, then perhaps you should get a network attached storage that has RAID . that way you can swap out hard disks.
if you want true archival lifetime , then you might have to print them. thats what the Smithsonian recommends. heh
maybe you should think about a commercial cloud company.
usb enclosures are the worst. unless you installed a driver, the usb enclosure blocks s.m.a.r.t. so while your drive may be failing, your operating system has no idea, because the drive isn't attached directly to its sata port.
i heard a lot of hard disks failing, and getting bad sectors, and eventually crashing. i never heard of magnetic / mechanical hard disk rot of data
yeah. if your photos are that important , i think you should just go ahead and pay apple the 9.99 a month for 1 terrabyte of iCloud until iCloud gets discontinued.
i don't think even iCloud will be around in 25 years
Last edited:
I've also heard a few rumblings that Blu-Ray discs are being used for archival cold storage now in some datacenters. I'm not sure how much different, if at all, they are to consumer grade discs, but it may be something worth looking into.
Optical discs are still a viable back-up solution. In my experience, optical discs are at least as reliable as an HDD, possibly more so if quality media is used, files are spot-checked manually on a completed burn, and the discs are stored properly. I work in the print industry. It is standard procedure to back up all old files to DVD. Before that we used CD-R. I access reference files that are five, ten, or even fifteen years old (these on CD-R) and it is very rare that we come across a disc that will not read. I recently came across a disc that was found to be corrupted. That may be the only disc I have ever come across in my career that hasn't worked. It is certainly the only one I can remember over the last twelve years. The disc predated my working here, so it is impossible to know if the burn was bad or the data had begun to rot. DVD also has an advantage in that the potential loss is only 5GB at a time. Disadvantages come with large data sets. Discs can be unwieldy to store and burning 5GB at a time may be more than most users are willing to undertake. I am thinking of stepping up to Blu-Ray for my personal files for this reason.
I use a combination of three or four back-up places for my photos. I have a few hundred GB stored in my Lightroom catalog. I also have a duplicate stored on a 3.5" external drive. Most of those photos are also stored on an older, additional 3.5" drive. I am slowly backing all this up to disc as a last resort. The key is to spread out the data out to as many places as possible. This does not eliminate the probability, but can reduce the likelihood of total loss to near ZERO percent.
I use a combination of three or four back-up places for my photos. I have a few hundred GB stored in my Lightroom catalog. I also have a duplicate stored on a 3.5" external drive. Most of those photos are also stored on an older, additional 3.5" drive. I am slowly backing all this up to disc as a last resort. The key is to spread out the data out to as many places as possible. This does not eliminate the probability, but can reduce the likelihood of total loss to near ZERO percent.
I'm not well versed in professional printing. I have often considered printing my top favorite 1%er's out in order to have non-digital record of them. I've made books of my kids pics but I doubt Apple's books are Smithsonian lifetime archival quality...maybe they are?
Is there a recommended print method for achieving the best possible non-digital archival record of a photo? I imagine, the ink, paper, DPI, size, and other criteria are important. Especially if you want the ability to scan the photo back in digitally at reasonable quality for some reason in the future.
I'll do some searching on the interwebs...just wondering if there is a standard.
I use a number of unlimited cloud backups such as CrashPlan and SOS backup for my ~2 TB picture archive. They are inexpensive, around $100 a year for unlimited storage.
For disk rot you can refresh your disks at the command line. Larry Jordan at larryjordan.com has an article about refreshing hard disk storage:
https://larryjordan.com/articles/technique-refreshing-hard-disk-storage/
at a terminal prompt you find your disks:
df -k
Filesystem 1024-blocks Used Available Capacity iused ifree %iused Mounted on
/dev/disk1 975902848 755810708 219836140 78% 189016675 54959035 77% /
devfs 185 185 0 100% 642 0 100% /dev
map -hosts 0 0 0 100% 0 0 100% /net
map auto_home 0 0 0 100% 0 0 100% /home
/dev/disk3 39056448 39056448 0 100% 18446744073690023769 19528224 4893035563313003520% /Volumes/CABARET
/dev/disk4s2 9765275600 8208098016 1557177584 85% 513006124 97323599 84% /Volumes/Promise Pegasus
/dev/disk2s1 3907018568 663717040 3243301528 17% 18446744072898725351 810826265 100% /Volumes/Oppo
So we've got disks at /dev/disk1, /dev/disk2, /dev/disk3, /dev/disk4. The extra letters (like s1, s2) are the partitions but since we want all we can ignore them.
Disk 3 is a blu-ray so we con't need that so we just need the boot disk (/), the Pegasus and Oppo.
sudo cat /dev/disk2 > /dev/null
will refresh the Oppo disk.
You can also find the disk names in disk utility.
But beware - you can also wipe the whole disk if you type this in wrong!
For disk rot you can refresh your disks at the command line. Larry Jordan at larryjordan.com has an article about refreshing hard disk storage:
https://larryjordan.com/articles/technique-refreshing-hard-disk-storage/
at a terminal prompt you find your disks:
df -k
Filesystem 1024-blocks Used Available Capacity iused ifree %iused Mounted on
/dev/disk1 975902848 755810708 219836140 78% 189016675 54959035 77% /
devfs 185 185 0 100% 642 0 100% /dev
map -hosts 0 0 0 100% 0 0 100% /net
map auto_home 0 0 0 100% 0 0 100% /home
/dev/disk3 39056448 39056448 0 100% 18446744073690023769 19528224 4893035563313003520% /Volumes/CABARET
/dev/disk4s2 9765275600 8208098016 1557177584 85% 513006124 97323599 84% /Volumes/Promise Pegasus
/dev/disk2s1 3907018568 663717040 3243301528 17% 18446744072898725351 810826265 100% /Volumes/Oppo
So we've got disks at /dev/disk1, /dev/disk2, /dev/disk3, /dev/disk4. The extra letters (like s1, s2) are the partitions but since we want all we can ignore them.
Disk 3 is a blu-ray so we con't need that so we just need the boot disk (/), the Pegasus and Oppo.
sudo cat /dev/disk2 > /dev/null
will refresh the Oppo disk.
You can also find the disk names in disk utility.
But beware - you can also wipe the whole disk if you type this in wrong!
So here is what the National Archives lists as the specs for a print "at an appropriate quality level to substitute for the original record if the original record copy is no longer viable". You can see specs for smaller source images at the bottom of the page.
http://www.archives.gov/preservation/products/products/spp-p3.html
I not interested in printing or saving my photos to DVD. I add to the collection I am most concerned to preserve the thousands of family photos I scanned a few years back. I do have my data backup to the cloud using crash plan. But what concerns me is if a file is sitting on the hard drive and has not been accessed in a few years I know it is possible for the sector to lose the charge and possible become un readable. If is goes bad on the master disk and everything is backed from there then count all the backups go bad also. So of the photos are stored as tiffs so reserving the could be a option to refresh the files, but others are jpegs so if I was to reserve them I guess I would need to save them as tiffs.
I found a site with some great tools for archiving. I am using Fixity to check the integrity of my files.
Check out this site and the tools...
https://www.avpreserve.com/avpsresources/tools/
Check out this site and the tools...
https://www.avpreserve.com/avpsresources/tools/
This is why you don't use "super duper or rsync" for backup. Let's say you have a good copy of some image. Now the OS does something and the file is corrupted but you don't know it. At this point you are still OK because you have the super duper or rsync backup. But then you do another backup and good backup file is over written with the corrupted version.
How to avoid the above. Use Apple's Time Machine. It keeps all the old versions of the files and tries not to over write old data. Your TM disk needs to be at least twice the size of the data you need to back up. For you with "only" 400GB of data even a 2TB disk drive would be enough.
You CAN use Super Duper safely but what you need to do is own a dozen disk drives and ROTATE them. Put a few on a weekly rotations and a few on a monthly and a few one a yearly cycle and then you can recover years old data if you want. This is what people running data centers 30 years ago did with tape. We's have a "tape vault" with hundreds of reels of tape. Much simpler to just buy a 2TB or even 6TB disk drive and connect it with TM and let it make hourly backups
If you had a LOT more data, I'd say keep it and backup copies on a FreeNAS server that is running ZFS because it does do data verification end to end and can make zero-cost snapshots of the file system so you can actually back up a ruling system by first snapshotting it then copying the snapshot. ZFS starts to make sense after 8TB or so of data but for your 0.4TB just buy a disk dive and use it for Time Machine as your primary backup system. Then do you off site with something like Crash Plan or BackBlaze
I've been using a 2 disk mechanical RAID system, with a third disk held off site, swapping out periodically.
Is there a significant downside to backing up my photos on my universities unlimited google drive storage? I am not using Google Photos, from what I understand Google Photos automatically degrades DSLR quality images.
I have this along with a local backup on an SSD. Some of them are archived as 7z with no compression.
I have this along with a local backup on an SSD. Some of them are archived as 7z with no compression.
The only other thing I can suggest is buying an internal hard drive (or 2) and an HDD docking station to make a 3rd or 4th backup to store in a bank safety deposit box.
That is where my triplicate backups of my home videos and photos go. I back up to the 2nd'ary backups once a week and then the triplicates every few months. This could also depend on how often once shoots as well.
I haven't used blu ray yet, but I'll be putting my home videos on that as well just as a paranoid other copy.
But here's the other question (and probably worthy of a seperate thread) - how do we manage everything? lol
For my photos, I use Aperture and it's well organized (someday maybe Lightroom or something else)
For my videos, I have a pile of Final Cut X archives for which I need to edit therefore eventually having a pile of videos. Those can go in iTunes, but it would be better (for me) to have them with the photos as those are chronologically saved.
Again, probably a whole other discussion lol
Cheers,
Keebler
Having your data spread as many places as possible is the best protection plan.
-Copy to DVDs (normal or BlueRay). Keep the discs in a good enclosure. Don't put them on top of each other to avoid scratches.
-If you have 400GB "only", save your pics on Flickr. You have 1TB for free and they don't compress your photos. Keep in mind that videos are limited to 3min.
-Save them to Google Photos too. Unlimited storage. They compress your photos but the quality is good. Hard to notice any difference.
Having Flickr or Google Photos in your phone is great since you'll have access to all your pics everywhere.
DVDs are cheap. Flickr and GP are free. See no reason to pay for cloud storage.
-Copy to DVDs (normal or BlueRay). Keep the discs in a good enclosure. Don't put them on top of each other to avoid scratches.
-If you have 400GB "only", save your pics on Flickr. You have 1TB for free and they don't compress your photos. Keep in mind that videos are limited to 3min.
-Save them to Google Photos too. Unlimited storage. They compress your photos but the quality is good. Hard to notice any difference.
Having Flickr or Google Photos in your phone is great since you'll have access to all your pics everywhere.
DVDs are cheap. Flickr and GP are free. See no reason to pay for cloud storage.