External Drive - Time Capsule - Time Machine Restoration

[JF117]

Hi,

Hopefully someone can help me out!

I used an external passport device [4tb] to back up my data via time machine and have been periodically backing up since Jan 2023. When I was conducting some maintenance on my MBA 2017 [which has the adapter / 2tb NVME M2 drive installed], after I reformatted the SDD, by complete mistake and through disk utility, I inadvertently restored the external drive which contained all my backups to one of its original snapshots [Jan 2023]. Stupid me, I know.

Bringing the external drive to a local data recovery specialist he told me that the drive was faulty from the beginning and the backups I thought I was writing onto the external drive were never backing up to begin with as each time it attempted a recovery, a ".crash" file apparently followed. I was devastated obviously. He also told me that the drive was 'on its last legs' and that any further scans I do, I should proceed with caution, and I have.

I decided to restore the only snapshot I could onto the 2TB SSD and begin piecing things back together, so I at least had a bit of a starting point.

After that, I used disk drill to conduct a deeper scan and it transpired all of my data was there, what a miracle! Since then, I have recovered most of the important files onto a 16TB IronDisk which I intend to use as part of a NAS Set Up down the line.

When D-Drill done its scan on the 4TB it pulled upward of about 15.5tb of data which I have come to learn is the distinction between logical and physical data, which I am still not 100% on the permutations of that. In any case, all my backups were there! I have attached some photos to demonstrate what I'm talking about and have since successfully recovered them onto the 16TB.

Not sure how this works but when I recovered the backups they were worth 12.5tb worth of storage which, as far as I know, is duplicated data where the recovery program does not distinguish between logical and physical so creates multiple copies of the same data instead of incremental ones? ( might be wrong here so feel free to correct )

Regardless however, my question to this forum is, assuming I have all the relevant backups, the data that comes with the backups and its intact, is there any way for me to reconstruct a time machine file that would allow me to restore my system the way I left it on 28-Jul-24?

In essence:

1. Is it possible?
2. What do I need to do?
3. Is there a third party software that can do this for me?

Thanks in advance for your help!

 

[Bigwaff]

is there any way for me to reconstruct a time machine file that would allow me to restore my system the way I left it on 28-Jul-24?

Was your TM drive formatted HFS+ or APFS? The version of TM and how it stores and references file data is dependent on its file system format and version of macOS being used. While you should be able to dive into the latest backup folder and copy files to another disk, you probably won’t be able to “reconstruct” the TM backups such that you will be able to use Time Machine app to restore files.

 

[HDFan]

In the future you might want to consider implementing a 3-2-1 backup plan with only 1 of those backups being TM.

 

[JF117]

Was your TM drive formatted HFS+ or APFS? The version of TM and how it stores and references file data is dependent on its file system format and version of macOS being used. While you should be able to dive into the latest backup folder and copy files to another disk, you probably won’t be able to “reconstruct” the TM backups such that you will be able to use Time Machine app to restore files.

It would have been APFS.

When I try and access or move them it comes up with an error message along the lines of 'cannot move time machine backup files' or something.

Any ideas on how to get around this?

In the future you might want to consider implementing a 3-2-1 backup plan with only 1 of those backups being TM.

Yes, I bought a 16tb NAS drive to set up down the line.

 

[HobeSoundDarryl]

TM is not meant to be used as you seem to be using it. TM is supposed to be like a "for dummies"-type backup solution where the Mac basically takes care of everything and users don't really do anything except turn it on and let it do its thing.

Occasionally when TM backups need re-creation, Mac tells you and you choose "yes" or "OK" and let it create a new backup. There shouldn't be any "periodically," as TM is automatic- just leave the drive connected and let it work. There shouldn't be any "restoring" of select snapshots but just let it work as "whole" backups and let it manage all snapshots itself. In short, you seem to be trying to inject wants into a system that is meant to work independent of such activities.

My advice:

1. If you want to mess around/control backups on more granular levels, seek out other options instead of TM. CCC & SuperDuper are both great options that can offer a lot of flexibility for those who want to be more involved in how things are backed up.

2. If you want to use TM, adopt how it is intended to be used, which is mostly staying OUT of the files and letting it take care of itself.

Further and for EITHER option, instead of leaning on a single backup drive, I suggest at least TWO drives for backups with one always stored offsite and regularly rotating with the one onsite so that the former is always a pretty fresh backup.

Adding just one more drive as a second backup also covers the scenario of "surprise!" faulty drive so that if one drive happens to fail at any time, you don't have to go spend money on data recovery- just replace the bad drive with a new one while the existing drive continues to act as backup. You'll have 2 "backups" at that time: one in the Mac (the original files) and one backup on the drive still working fine.

Once the replacement drive arrives, backup to it, get one of the two offsite and you are again well protected for just about all scenarios. Big storage is dirt cheap. Buy big storage in at least 2 drives and add the fresh offsite backup drive to protect against common loss scenarios like fire, flood and theft (which often takes out ALL storage at one location).

With 3 copies of files: Mac + backup 1 + backup 2, you'll probably never face having to go deal with data recovery specialists again. If that would STILL be a worry, buy ONE more drive for backups so you have 2 copies on site and 2 offsite... and let the latter be stored at DIFFERENT offsite locations. And if TM gives you any pause as a system, mix in CCC and/or SuperDuper backups so you are not putting all backup eggs in one (TM) basket.

Lastly: if you have old drives that you have mostly retired but still functional, back up once to them so you have a few more backups laying around on drives you already own. Core stuff like a music collection, photos, etc don't change over long timespans (other than additions of new files), so converting "retired" drives to archive storage use means you have a few more ways to recover MOST of what you own. And if you refresh them regularly with the new music, new photos, etc, they will be relatively fresh if you needed to go to them for recovery.

 

[JF117]

TM is not meant to be used as you seem to be using it. TM is supposed to be like a "for dummies"-type backup solution where the Mac basically takes care of everything and users don't really do anything except turn it on and let it do its thing.

Occasionally when TM backups need re-creation, Mac tells you and you choose "yes" or "OK" and let it create a new backup. There shouldn't be any "periodically," as TM is automatic- just leave the drive connected and let it work. There shouldn't be any "restoring" of select snapshots but just let it work as "whole" backups and let it manage all snapshots itself. In short, you seem to be trying to inject wants into a system that is meant to work independent of such activities.

To be clear, this drive was used to backup a laptop as I travelled and was connected and backed up on an ad hoc basis, hence why they were periodic.

I'm having to do this because I have wrecked my TM backups by restoring the TM / External drive back to its original and first snapshot [Jan-23], ergo, "deleting" all subsequent backups which took place after and up to the last back up [Jul-24].

Using recovery software I was able to recover them all but they're without the relevant 'pointers' and other 'metadata' which, as far as I know, permits them to be read as TM backups upon plugging it in.

That's why I'm attempting to 'reconstruct' as ideally, instead of manually trawling through file directories for the files I want, with a bit of labour I could possibly and simply restore it back onto my laptop the way it was left. Ultimately, that's my question here, is this possible?

My advice:

1. If you want to mess around/control backups on more granular levels, seek out other options instead of TM. CCC & SuperDuper are both great options that can offer a lot of flexibility for those who want to be more involved in how things are backed up.

2. If you want to use TM, adopt how it is intended to be used, which is mostly staying OUT of the files and letting it take care of itself.

Again to be clear, this is generally how I use TM but what I want to know is if there is anything out there which will let me reconstruct the recovered backups so they can be read and utilized by TM.

Further and for EITHER option, instead of leaning on a single backup drive, I suggest at least TWO drives for backups with one always stored offsite and regularly rotating with the one onsite so that the former is always a pretty fresh backup.
Adding just one more drive as a second backup also covers the scenario of "surprise!" faulty drive so that if one drive happens to fail at any time, you don't have to go spend money on data recovery- just replace the bad drive with a new one while the existing drive continues to act as backup. You'll have 2 "backups" at that time: one in the Mac (the original files) and one backup on the drive still working fine.

Once the replacement drive arrives, backup to it, get one of the two offsite and you are again well protected for just about all scenarios. Big storage is dirt cheap. Buy big storage in at least 2 drives and add the fresh offsite backup drive to protect against common loss scenarios like fire, flood and theft (which often takes out ALL storage at one location).

With 3 copies of files: Mac + backup 1 + backup 2, you'll probably never face having to go deal with data recovery specialists again. If that would STILL be a worry, buy ONE more drive for backups so you have 2 copies on site and 2 offsite... and let the latter be stored at DIFFERENT offsite locations. And if TM gives you any pause as a system, mix in CCC and/or SuperDuper backups so you are not putting all backup eggs in one (TM) basket.

Lastly: if you have old drives that you have mostly retired but still functional, back up once to them so you have a few more backups laying around on drives you already own. Core stuff like a music collection, photos, etc don't change over long timespans (other than additions of new files), so converting "retired" drives to archive storage use means you have a few more ways to recover MOST of what you own. And if you refresh them regularly with the new music, new photos, etc, they will be relatively fresh if you needed to go to them for recovery.

The set up which I employed was suffice, it wasn't the set up or 'faulty' drive which failed me it was my own carelessness when carrying out certain tasks in disk utility but yes, this is what I plan to do.

 

[Bigwaff]

When I try and access or move them it comes up with an error message along the lines of 'cannot move time machine backup files' or something.

Any ideas on how to get around this?

The error is when you attempt to copy what look like "recovered" TM files from the 16TB IronDisk? The 16TB IronDisk is formatted APFS? I'm afraid you're in for a learning curve and use of command line tools. Here is an explainer of APFS TM backups

Time Machine to APFS: Backup structure and access

Unlike HFS+ backups, those on APFS volumes look different depending on how you try to access them. What you see in the Finder is an illusion not available to other apps.

eclecticlight.co

It's difficult to know whether your "recovered" TM folders will appear as true TM backup snapshots to utilities that understand TM backup snapshots. What is clear is you can't use the Finder like you could with HFS+ TM backups. You basically have to mount the TM backup snapshot to access files. In the article, there is a mention of using Carbon Copy Cloner to mount TM backup snapshots. If you Google something like "mount time machine backup snapshot", you might find other possible techniques.

Caveat: Can't tell you if any of this will work. Have never tried myself, especially under your unique circumstances. Just offering suggestions.

Best of luck.

 

[JF117]

The error is when you attempt to copy what look like "recovered" TM files from the 16TB IronDisk? The 16TB IronDisk is formatted APFS? I'm afraid you're in for a learning curve and use of command line tools. Here is an explainer of APFS TM backups

Time Machine to APFS: Backup structure and access

Unlike HFS+ backups, those on APFS volumes look different depending on how you try to access them. What you see in the Finder is an illusion not available to other apps.

eclecticlight.co

It's difficult to know whether your "recovered" TM folders will appear as true TM backup snapshots to utilities that understand TM backup snapshots. What is clear is you can't use the Finder like you could with HFS+ TM backups. You basically have to mount the TM backup snapshot to access files. In the article, there is a mention of using Carbon Copy Cloner to mount TM backup snapshots. If you Google something like "mount time machine backup snapshot", you might find other possible techniques.

Caveat: Can't tell you if any of this will work. Have never tried myself, especially under your unique circumstances. Just offering suggestions.

Best of luck.

Yes, I was already going through that very weblink before you mentioned it but yeah, that's the crux of the matter.

I'll continue to have a look at that, thanks!

 

[gilby101]

It would have been APFS.

Base on your screenshot which shows folders (and not TM/APFS snapshots), the TM disk is formatted HFS+.

What version of macOS?

When (in Finder) you select one of the folders with completed backups (the folder name ends in .backup), what do you see? If it is not in error, you should be able to delve down into any complete backup and recover (by drag and drop) folders and files to another drive.

Having so many incomplete, interrupted, etc. backups is definitely a worry. Probably caused by your disconnecting the drive before a backup was finished. But I would hope that the earlier complete backups are OK.

Bringing the external drive to a local data recovery specialist he told me that the drive was faulty from the beginning and the backups I thought I was writing onto the external drive were never backing up to begin with as each time it attempted a recovery, a ".crash" file apparently followed. I was devastated obviously. He also told me that the drive was 'on its last legs' and that any further scans I do, I should proceed with caution, and I have.

Does your "local recovery specialist" have decent macOS knowledge? Your description leaves me doubting it.

When D-Drill done its scan on the 4TB it pulled upward of about 15.5tb of data which I have come to learn is the distinction between logical and physical data

Don't be surprised by the size. Disk Drill is probably recovering the same file from each backup.

 

[JF117]

Base on your screenshot which shows folders (and not TM/APFS snapshots), the TM disk is formatted HFS+.

What version of macOS?

When (in Finder) you select one of the folders with completed backups (the folder name ends in .backup), what do you see? If it is not in error, you should be able to delve down into any complete backup and recover (by drag and drop) folders and files to another drive.

Having so many incomplete, interrupted, etc. backups is definitely a worry. Probably caused by your disconnecting the drive before a backup was finished. But I would hope that the earlier complete backups are OK.


Does your "local recovery specialist" have decent macOS knowledge? Your description leaves me doubting it.


Don't be surprised by the size. Disk Drill is probably recovering the same file from each backup.

It was the latest Monterey

The folder names dont end with .backup, they're just folders with the expected directories. I only used it for music and photos and can go right down to each of these respective file paths with the majority of files there, intact.

TBH, for other reasons, the recovery specialist left me feeling a bit shafted but primarily because he emphasized any recovery efforts would be futile/pointless and that led to other decisions which inadvertently cost me £.

I've found a ".previous" folder with approx 1.55tb of data in it and taken on 28-Jul-24, which is as close to the size of the original partition as I think it will get and my last known backup.

The ".previous" folder, that would be a fairly complete backup, right?

 

[gilby101]

I've found a ".previous" folder with approx 1.55tb of data in it and taken on 28-Jul-24, which is as close to the size of the original partition as I think it will get and my last known backup.

The ".previous" folder, that would be a fairly complete backup, right?

A bit worrying that there are TWO .previous folders. But try them or any of the .backup folders which should be complete backups at those dates.

I only used it for music and photos and can go right down to each of these respective file paths with the majority of files there, intact.

Get back what you can onto another disk. Much easier to use Finder (if you can) than resort to recovery tools like Disk Drill which are intended for when the folder structure has been destroyed.