Offsite backup strategy failed - now what?

[rawdawg]

thanks for those thoughtful comments. Just to clarify, I did have an onsite and offsite backup (just not an active version because this was archived and I don't have a RAID storage setup---with that much data on a mac mini it's not feasible to keep that many drives up any other way)

Since reading Tesselator's first response and waiting to hear what clone unit he has I realized CCC has a block-level copy function and have been attempting to clone it that way. It failed ultimately failed 3 times due to errors using my dock, but my handy old Newertech adapter is working, except it's been going 11 hours and still looks like it's near the beginning.

 

[rawdawg]

...after that image is created and safe if there is still something you need to recover off the broken drive you will know where it is by examining the clone's map (which I would keep locked BTW - so that your computer doesn't try to write to it or "fix it" while you're studying the structure). SpinRite can then be used on that specific area (of the broken drive) to try and recover what's left....

I suspected if I knew where on my hard drive certain files lived, then I could allocate time in SpinRite to work on those areas and finally retrieve the data I need.

You mention doing this on the clone I make from the bad drive. But I can't find anywhere online that explains how or where to find the info on where that data may live within your HDD! I think SpinRite would require it to be in a percentage of the HDD because that's the only way I know how to start the program anywhere other than the beginning.

Can you tell me what you use to find this map!? Thank you so very much

 

[d-m-a-x]

OP, sorry to hear about your loss.

Two subsections:
1st off, HD failures:

Hard drive failures come in different guises, and the remedies are (likewise) different. Thus the first step (after noticing that something is wrong) is trying to ascertain what is wrong. Wikipedia's article is long-winded and lacks any how-to's, but may be worth a read. (http://en.wikipedia.org/wiki/Hard_disk_drive_failure)

Basically: If your computer sees the drive (and reports it's capacity correctly), and the drive is not making any frightening noises, the data might be recoverable by using specialist software. If not, the problem is either mechanical (head crash or mechanic failure) or the controller circuitry is screwed. Unless you have a healthy disk from the same batch lying around, both necessitate a third-party specialist. If your data is economically valuable or your business has insurance, this alternative is worth exploring.

2nd, backup&redundancy - policies

The thing with hard drives is that they mostly do not fail, but when they fail everybody's cursing. Back at the change of the millennium I was working at a small IT company, and our main file server was based on a RAID 5 -array of IBM GXP75's - (see http://hexus.net/tech/news/storage/209-ibm-gxp75-failures/ ) - man was that a nightmare, we basically came to the office in the morning and every second day one of the GXP's had failed over night so we went through some 30 drives before getting a totally new solution...

Anyway,
In my experience (both in offices and home offices), I've found that a three tier-solution offers quite a good balance.

1st tier: Local machine (workstations / file servers)
2nd tier: Local high-availability backup
3rd tier: Offsite backup.

Off course, if you're serious about your data you need an offsite backup. But whatever your offsite backup is, it will not be easy to access. That's where the second tier comes in: The purpose of the second tier is to protect against machine disasters, drive failures as well as "dumb users". Thus it needs to used every now and then and must be easy to access...

It also helps if the second tier is in itself both reliable and redundant.

I agree with d-m-a-x, that unless we're talking about data-center volumes, optical disks offer a viable route - they are both cheap and easy to transport and store and allow you to easily offsite-backup in multiples (always burn two disks and store them in separate sets). The caveat is that if you have no "library management" (for the lack of a better word), you'll be in a pinch when you're trying to ascertain which disc contains the most recent backups of files.

I recently posted a poll here, but did not get very many answers... https://forums.macrumors.com/threads/1539302/

My main interest in posting the poll (which I did not want to mention as I was afraid it might be suggestive) is that considering the amounts of data many of us are working with, the concept of internet-based daily backups are hitting the limits of feasibility (data throughput rates, especially on asynchronic lines).

RGDS,

For the most part, i'm lucky that my jobs have a beginning and an end. Once the job is done there is a good chance i won't touch it again. This makes backing up easy, because once i burn a couple of discs (1 hero and 1 raw) - it gets erased from the hard drive and put in storage. Sometimes there will be a big, ongoing one - in which case i case, number the files, make a low rez catalog, burn the raw to disc and pull files as needed

 

[Tesselator]

I suspected if I knew where on my hard drive certain files lived, then I could allocate time in SpinRite to work on those areas and finally retrieve the data I need.

You mention doing this on the clone I make from the bad drive. But I can't find anywhere online that explains how or where to find the info on where that data may live within your HDD! I think SpinRite would require it to be in a percentage of the HDD because that's the only way I know how to start the program anywhere other than the beginning.

Can you tell me what you use to find this map!? Thank you so very much

After you make the clone you can learn which files are whole and which didn't make it. From there you have their file names. From the file names you can find their start and follow their chain (maybe) if they were fragmented. I guess there are a lot of tutorials on-line for doing this. It's kind of beyond the scope of my interest to tutorialize it here. Sorry bro. But google works. I got results using these strings:

• OS X finding drive sector by filename
• HFS finding drive sector by filename
• Understanding the HFS format
• HFS Recovering deleted files the hard way
• HFS Sector level file recovery
• Finding a files allocated blocks in HFS
• HFS catalog file

But I do this kind of sorting and sifting a lot on the subjects of both world politics and computer science so for me it goes really fast - YMMV. For example searching for "HFS catalog file" returned this as one of the results: http://libfslibs.googlecode.com/files/Hierarchical File System (HFS).pdf and I could immediately scroll to this section:

7. The catalog file

The catalog file is a B-tree file used to maintain information about the hierarchy of files and directories of a volume.

The allocation block number of the first file extent of the catalog file (the header node) is stored in the master directory block (HFS) or the volume header (HFS+). The B-tree structure is described in section 3 B-tree file.

Each node in the catalog file is assigned a unique catalog node identifier (CNID). The CNID is used for both directory and file identifiers. For any given file or directory the parent identifier is the CNID of the parent directory. The first 16 CNIDs are reserved for use by Apple and include the following standard assignments:

page 28​

Etc.
With the next 50 pages detailing it...

It sounds difficult but it's not really. So it's either waiting over 8 months for SpinRite to finish the entire drive (and probably killing the drive all together) or studying for a few hours.

As for which tools to use there's a lot you can do just from the OS and there's lots of free tools you could assemble into a suite. I like the commercial app called FileXray quite a lot. You can read about it online here: http://filexray.com/fileXray.pdf and just scrape off that filename for their web page.


.

 

[rawdawg]

It's kind of beyond the scope of my interest to tutorialize it here..... But google works. I got results using these strings:

• OS X finding drive sector by filename
• HFS finding drive sector by filename
• Understanding the HFS format
• HFS Recovering deleted files the hard way
• HFS Sector level file recovery
• Finding a files allocated blocks in HFS
• HFS catalog file

If you can believe me, I tried every combination I could think of and never found anything!! I was mainly searching words like:

• How to data map hard drive
• Where on hard drive sector is a file
• ...

and search perimeters like that. So I VERY much appreciate you providing me with correct search terms - I was literally getting nothing.

And though you say it's beyond your interest to tutorialize, the fact you've taken the time to write what you have when you could have just put a quick one sentence reply shows just how helpful you are. You've done more than enough to help me, thank you.

 

[Mikey7c8]

If the data is important, may be worth shelling out for a recovery company to do it for you.

Last time my company had to do it, it cost something like $2k - but we got the data back. Something to think about perhaps.

 

[Tesselator]

If you can believe me, I tried every combination I could think of and never found anything!! I was mainly searching words like:

• How to data map hard drive
• Where on hard drive sector is a file
• ...

and search perimeters like that. So I VERY much appreciate you providing me with correct search terms - I was literally getting nothing.

And though you say it's beyond your interest to tutorialize, the fact you've taken the time to write what you have when you could have just put a quick one sentence reply shows just how helpful you are. You've done more than enough to help me, thank you.

Sure, NP, you're welcome!

The main reason I don't wanna tute it is cuz the process is too dynamic and involved. The basic idea is simple-ish but that's already out there - in spades. Or used to be anyway, I assume still... All that's needed is to be pointed in the right direction. So all there is to do now is read up on like 3 or 4 structures (HFS "file" formats) and then go through the tedious steps of location and recovery attempts. It's actually kinda fun the first time - or was for me. There's usually not toooo many wrecked files - although there can be...

 

[nigelbb]

i have two blu ray burners. once a month i burn a spindle of discs, then keep them in a dark drawer (important). Been doing this for 15 years (since cd's). Have never had a problem with lost data.

Goodness! How many disks? How long does that take you? External hard disks are now so cheap that it doesn't seem worth it. I just bought a Hitachi 4TB external disk for £140 ($215) including 20% sales tax. 4TB is the equivalent of about 150 Blu-ray disks. Just fill it up & put it in a drawer & start filling up another.

 

[Tesselator]

Yup, DVD backup is great when there's less than 4 or 5 disks worth. Or BR with 2 or 3 disks. Past that it's just too time consuming!

d-m-a-x,
I know a guy who never makes backups ever, on anything, and has never in 15 years had a problem with data loss. I'm not sure such anecdotal examples are meaningful or useful - for when the beast rears it's fugly head you're either prepared or you're not. And the fugly futher mucker comes in all sorts of incarnations including house fires, floods, burglary, blue ice, solar flares, user stupidity (human error), engineered failure rates, device agedness, and the list goes on. A spindle-full of BR disks might work for you (with luck) but no one would seriously recommend that. You know that right?

 

[nigelbb]

If the data is important, may be worth shelling out for a recovery company to do it for you.

Last time my company had to do it, it cost something like $2k - but we got the data back. Something to think about perhaps.

+1 These guys can take the drve apart & do stuff that consumer software is just incapable of.

 

[d-m-a-x]

Goodness! How many disks? How long does that take you? External hard disks are now so cheap that it doesn't seem worth it. I just bought a Hitachi 4TB external disk for £140 ($215) including 20% sales tax. 4TB is the equivalent of about 150 Blu-ray disks. Just fill it up & put it in a drawer & start filling up another.

about 20 disks a month - 1 disc per job. It is my pref that all the jobs are not on one single drive. I hear too many stories of a drive going bad and loss of multiple projects, no thanks

----------

Goodness! How many disks? How long does that take you? External hard disks are now so cheap that it doesn't seem worth it. I just bought a Hitachi 4TB external disk for £140 ($215) including 20% sales tax. 4TB is the equivalent of about 150 Blu-ray disks. Just fill it up & put it in a drawer & start filling up another.

Yup, DVD backup is great when there's less than 4 or 5 disks worth. Or BR with 2 or 3 disks. Past that it's just too time consuming!

d-m-a-x,
I know a guy who never makes backups ever, on anything, and has never in 15 years had a problem with data loss. I'm not sure such anecdotal examples are meaningful or useful - for when the beast rears it's fugly head you're either prepared or you're not. And the fugly futher mucker comes in all sorts of incarnations including house fires, floods, burglary, blue ice, solar flares, user stupidity (human error), engineered failure rates, device agedness, and the list goes on. A spindle-full of BR disks might work for you (with luck) but no one would seriously recommend that. You know that right?

guess you dont know unless you try.

 

[d-m-a-x]

Yup, DVD backup is great when there's less than 4 or 5 disks worth. Or BR with 2 or 3 disks. Past that it's just too time consuming!

d-m-a-x,
I know a guy who never makes backups ever, on anything, and has never in 15 years had a problem with data loss. I'm not sure such anecdotal examples are meaningful or useful - for when the beast rears it's fugly head you're either prepared or you're not. And the fugly futher mucker comes in all sorts of incarnations including house fires, floods, burglary, blue ice, solar flares, user stupidity (human error), engineered failure rates, device agedness, and the list goes on. A spindle-full of BR disks might work for you (with luck) but no one would seriously recommend that. You know that right?

Taking a couple hours to burn some discs is nothing compared to recovering data from a damaged drive - you know that, right?

 

[nigelbb]

Even using two burners in parallel I don't see how you can burn 20 BD disks in two hours especially with all the manual intervention & shuffling about of blanks & labelling burned disks.

For this volume of data you would be much better off using tape. The backup would be quicker less prone to error & the media is far more reliable.

 

[d-m-a-x]

Even using two burners in parallel I don't see how you can burn 20 BD disks in two hours especially with all the manual intervention & shuffling about of blanks & labelling burned disks.

For this volume of data you would be much better off using tape. The backup would be quicker less prone to error & the media is far more reliable.

I can burn 20 disks in about 2 hours and I do

Tape? hahahahaha
DDS-4 tape is 19.8/gb per HOUR. A DDS-4 tape cost $7.00 as compared to a $1.00 blu ray disc.

Stick a fork in me, i'm done.

 

[Tesselator]

Taking a couple hours to burn some discs is nothing compared to recovering data from a damaged drive - you know that, right?

Of course.

But sitting in front of the machine, feeding it disks every 10min., labeling, and filing them for two or three hours straight isn't fun for me.

On the other hand there's nothing to just connecting an external drive or plugging one into a cradle dock, and letting it copy while you continue using your system or go about your other business.

Also in the time it takes to fill two or three BR disks I can back up terabytes of data over a single eSATA connection. And... at 20 BR disks a month that's the price of a 3TB HDD every 4 or 5 months.

But I guess everyone already knows these things and we didn't really need to talk about it again here... No?

 

[d-m-a-x]

Of course.

But sitting in front of the machine, feeding it disks every 10min., labeling, and filing them for two or three hours straight isn't fun for me.

On the other hand there's nothing to just connecting an external drive or plugging one into a cradle dock, and letting it copy while you continue using your system or go about your other business.

Also in the time it takes to fill two or three BR disks I can back up terabytes of data over a single eSATA connection. And... at 20 BR disks a month that's the price of a 3TB HDD every 4 or 5 months.

But I guess everyone already knows these things and we didn't really need to talk about it again here... No?

i often find the spindles for $30. 25gb x 50 discs is 1.25 tereabytes - pound for pound it's cheaper than hard drives. The main reason i do this is because when a drive fails you lose everything. By keeping a consistent backup on multiple pieces of media, you have less eggs in one basket. Tedious, yes... but i would rather put the work in and keep my data for later use rather than lose it.

----------

Of course.

But sitting in front of the machine, feeding it disks every 10min., labeling, and filing them for two or three hours straight isn't fun for me.

On the other hand there's nothing to just connecting an external drive or plugging one into a cradle dock, and letting it copy while you continue using your system or go about your other business.

Also in the time it takes to fill two or three BR disks I can back up terabytes of data over a single eSATA connection. And... at 20 BR disks a month that's the price of a 3TB HDD every 4 or 5 months.

But I guess everyone already knows these things and we didn't really need to talk about it again here... No?

I'm not saying this is what people should do, just that it's what i do. It's cheap and it works. I guess a lot of people drank the "optical media is dead" kool-aid that apple served up

 

[nigelbb]

I can burn 20 disks in about 2 hours and I do

Tape? hahahahaha
DDS-4 tape is 19.8/gb per HOUR. A DDS-4 tape cost $7.00 as compared to a $1.00 blu ray disc.

Why consider using 1999 technology? I can write to my LTO-5 tape drive at well over 100MB/s (faster than a hard disk). Each tape holds 1TB & costs the equivalent of about $40. Older generation drives can be found very cheap on eBay & the write speeds are many times faster than DDS-4 (3.2MB/s) e.g. LTO-1 is 20MB/s while LTO-2 is 40MB/s.

 

[Tesselator]

I used to use tape. It was fast, like you're saying. But the error there is that modern HDDs are 200MB/s or more - not less than 100. Just thought I'd mention that.

I liked tape backups - fun stuff.

I could never force myself to burn a bunch of disks tho. That'd be like Commodore 64 hell with the heat turned up. UG! And I assume if one iis paying $30 for 50 disks instead of the usual $65 to $85 then it's unsorted low QC media and not good for backups in the first place. No?

I don't think optical media is dead. But it was never good for backing up stuff. Maybe if it's "Playable Media" in restored or original format like music CDs Movie DVDs would be OK for a "collection". Or even BR films to BR disks... But as a backup system for computer files much over a few disks it fails. IMO anyway.

 

[d-m-a-x]

I used to use tape. It was fast, like you're saying. But the error there is that modern HDDs are 200MB/s or more - not less than 100. Just thought I'd mention that.

I liked tape backups - fun stuff.

I could never force myself to burn a bunch of disks tho. That'd be like Commodore 64 hell with the heat turned up. UG! And I assume if one iis paying $30 for 50 disks instead of the usual $65 to $85 then it's unsorted low QC media and not good for backups in the first place. No?

kodak bd-r

 

[Tesselator]

kodak bd-r

Cool! It's getting cheaper and cheaper. Looks like it's almost affordable now then - where you are. Here it's still about $2 a disk.

 

[nigelbb]

I used to use tape. It was fast, like you're saying. But the error there is that modern HDDs are 200MB/s or more - not less than 100. Just thought I'd mention that.

Modern tape drives have performance on a par with modern disks. So just as with backing up to an external disk provided the job completes in a reasonable time after you pressed the 'Go' button then it's fine as you are not having to change discs in a burner every 25GB. The advantage of tape versus an external disk is that tape is a genuine archival media & designed as such. You will have no problems reading the data even after i has been sitting on a shelf for years or even decades.

 

[Tesselator]

Modern tape drives have performance on a par with modern disks. So just as with backing up to an external disk provided the job completes in a reasonable time after you pressed the 'Go' button then it's fine as you are not having to change discs in a burner every 25GB. The advantage of tape versus an external disk is that tape is a genuine archival media & designed as such. You will have no problems reading the data even after i has been sitting on a shelf for years or even decades.

Agreed. Well, except for the last sentence. Tape is particularly vulnerable to background radiation and cosmic rays. It degrades over time probably faster than BR or DVD media. A lot depends on where you live. If you live in Boulder Col. where the BG radiation is the high I guess you better check them about once a year or so if you intend yo keep them longer. Cosmic ray events are spread in a gradient from our equator out as most of them spawn from the center of our galaxy - which is roughly on-edge to our equator. There was a guy (I believe it was here) some years back that worked at a data center of some kind, who told us how important /they/ felt that the tapes had to be checked once a month. I'm extrapolating that to a year for us common folk.

I know I sure wouldn't trust my family photo album to tape alone.

 

[seveej]

i have two blu ray burners. once a month i burn a spindle of discs, then keep them in a dark drawer (important). Been doing this for 15 years (since cd's). Have never had a problem with lost data.

There was a guy (I believe it was here) some years back that worked at a data center of some kind, who told us how important /they/ felt that the tapes had to be checked once a month.

This is another issue...

Just some months ago I had picked up an USB FDD drive (it came with a heap of some other junk), and I decided to see how well I could "recover" some data from my old Floppy disks (Amiga, PC). I had these four shoeboxes up in the attic and had really very little expectations of being able to recover much - the disks were maybe 18 y.o. on average (range: 12-27 y.o.) and for the last 10 years they had been in rough weather (range -35 - +40 centigrade).

Guess what: out of 454 disks only 1 disk was busted (i guess physically) and the rest were read cleanly. Most of them contained checksumed archive files so I could ascertain (with fairly high certainty) that the data was "clean". I guess this has something to do with the fairly low density of FDD's...
I then burned the entire heap onto DVD's (with duplicates), catalogued the contents and shippped them into storage.

I've had less luck with some of the earlier CD's I've burned in the mid 90's.

So I guess my question is (considering how "dumb" most CD/DVD readers and operating systems are), is there a consumer-grade solution for searching after CD/DVD-r degradation (so that you could make a new duplicate before the archive fails)?

 

[d-m-a-x]

This is another issue...

Just some months ago I had picked up an USB FDD drive (it came with a heap of some other junk), and I decided to see how well I could "recover" some data from my old Floppy disks (Amiga, PC). I had these four shoeboxes up in the attic and had really very little expectations of being able to recover much - the disks were maybe 18 y.o. on average (range: 12-27 y.o.) and for the last 10 years they had been in rough weather (range -35 - +40 centigrade).

Guess what: out of 454 disks only 1 disk was busted (i guess physically) and the rest were read cleanly. Most of them contained checksumed archive files so I could ascertain (with fairly high certainty) that the data was "clean". I guess this has something to do with the fairly low density of FDD's...
I then burned the entire heap onto DVD's (with duplicates), catalogued the contents and shippped them into storage.

I've had less luck with some of the earlier CD's I've burned in the mid 90's.

So I guess my question is (considering how "dumb" most CD/DVD readers and operating systems are), is there a consumer-grade solution for searching after CD/DVD-r degradation (so that you could make a new duplicate before the archive fails)?

the tech goes deep... from my experiance, the discs that have the layer on the top, rather than the sandwich deal where the layer in the middle of two discs glued together are better. There are different grades, and many internet forums who discuss them. I like the more common brands, kodak, staples, ritek, optical quantem, etc. It is important to verify (or at least check the first, middle, last files) for occasionally there will be a bad spindle of discs.

Bg radiation does not affect discs in my experiance, but it is very important to store them in a dark place, light makes them fade. oxidation is also a problem, i like sleevs vs cases. Heat is another problem, i tried to recover data for a friend who left his dvd booklet in the hot car and there was not one good disc. 90% of my discs are still good, and even the bad ones just have a few bad sectors, can still pull most of the data off them and reburn.

i dont know when they go bad, some of them seem to start around 5 years depending on the conditions

 

[nigelbb]

Agreed. Well, except for the last sentence. Tape is particularly vulnerable to background radiation and cosmic rays. It degrades over time probably faster than BR or DVD media. A lot depends on where you live. If you live in Boulder Col. where the BG radiation is the high I guess you better check them about once a year or so if you intend yo keep them longer. Cosmic ray events are spread in a gradient from our equator out as most of them spawn from the center of our galaxy - which is roughly on-edge to our equator. There was a guy (I believe it was here) some years back that worked at a data center of some kind, who told us how important /they/ felt that the tapes had to be checked once a month. I'm extrapolating that to a year for us common folk.

I know I sure wouldn't trust my family photo album to tape alone.

Magnetic tape does not suffer damage from cosmic rays. That idea is utter tosh. Cosmic rays are charged particles they don't interfere with magnetic particles on tapes. Computer memory may rarely be altered by cosmic rays which is one reason why servers use ECC memory. In the old days of reel to reel tape it was good practice every year or 2 that the tape was de-spooled & re-spooled to prevent data 'burning through' but better design of more recent cartridge tapes means this is no longer necessary.