Messed up encrypted APFS volume

[Taipan]

Hi!
I just made a mess. I got a new external DAS (Terramaster D4-320) for four disc drives connected to my Mac Mini. The four drives worked perfectly in another DAS before.
I don't know what I did, maybe the new DAS is crap, or maybe I accidently hot swapped the drives when troubleshooting some HD sleep issues.
Edit: In hindsight, I think what I did was: I hot swapped the position of the two drives in the enclosure, but prior to that I only ejected the volumes in Disk Utility, not the complete drives.

My problem now:
I have two drives with one encrypted APFS volume each, one is called "Time Machine", the other "Downloads". But now for some reason, BOTH of them show up as "Time Machine" in Disk Utility. And the one that should be called "Downloads" doesn't activate anymore. At startup, I get an error message saying "The volume 'Time Machine' cannot be unlocked. An error occured that prevents it from being unlocked" (roughly translated). It doesn't get to the point where I'm asked for the decryption key.
At some point, mac OS seems to have confused the drives and possibly wrote data to the Downloads drive, thinking it was the Time Machine drive.

I don't have high hopes for rescuing data from an encrypted volume, but maybe there's a chance that there's an expert here that can help me. There are about 4TB of data on the drive that I don't consider essential and are thus not backed up. Nevertheless I'd rather not lose them. :-S

I put the wrongly labelled drive back in to the old enclosure, which doesn't change anything, as expected.

Another thing that strikes me:
In Disk Utility, the wrongly labelled drive / Volume says "APFS-Volume - APFS (case sensitive, encrypted). I'm pretty sure the original "Downloads" volume was not formatted as case sensitive. I guess that mac OS uses case sensitive APFS for Time Machine, so this info seems to be taken over from the other drive as well.

Any help would be greatly appreciated!
Thank you!

 

[Taipan]

Some additional info:
diskutil shows the physical /dev/disk7 with a size of 8TB and /dev/disk7s2 as "Apple APFS container disk8" with a size of 8TB as well. This is correct.
Then for /dev/disk8 it shows an "APFS container scheme" with a size of 6TB. This is wrong, that's the size of the actual Time Machine container on the other disk.
Then it shows /dev/disk8s2 as an APFS volume "Time Machine" with a size of 3.1TB. I don't know where that number comes from.
There should be a single container and volume "Downloads" with a size of 8TB.

 

[DeltaMac]

Notice that your disk8 is not a physical disk, but is a synthesized (virtual) disk. And the Physical Store for disk 8 is on disk7.
disk7 is the physical disk, disk 8 is actually a synthesized disk. I think of it as a VM for the actual storage.(Don't know if that is an accurate use of "VM", but is simply how I look at that relationship with actual data storage
It adds to the confusion, when you only show the one physical disk, and the synthesized disk that would be software-associated with that physical disk. Only one is the physical disk.
(Might be better to show the full results of your diskutil list, as it may help you determine what is on each of the various types of disks, as displayed in that diskutil list.

 

[Taipan]

I'm aware of disk8 being a "virtual" representation of the physical storage container on disk7. That's why I posted these two, because together they represent the drive that I'm having problems with, and while the data for the physical drive seems to be correct, the data given for the "virtual" drive seems to be wrong. I don't understand why the other drives would be of any interest, unless you are referring to the "actual" Time Machine drive for comparison.
At the moment, I have unmounted the Time Machine drive out of fear to cause further "confusion", although that might be silly. But if it helps, I can mount it and provide the complete output.
Edit:
Here you go:

Disk 5/6 is the actual Time Machine drive, disk 9/10 is the one that should actually be Downloads. They look identical, with the exception of the physical size of the drive. I guess the partition data on the Downloads drive has been overwritten with the one from the Time Machine drive.
(I now realized where the 3.1TB comes from, apparently the size given is only the used space on the APFS volume).

Another thing I noticed: When the "false" Time Machine drive is connected at startup, I cannot mount the real Time Machine volume. I have to mount that one first.

 

[Taipan]

In case anyone is interested: I tried multiple recovery programs (Disk Drill, UFS Explorer, iBoySoft DataRecovery). UFS Explorer seemed the most promising, so after some fiddling, I contacted its support and described my problem. They were pretty friendly and said it would take manual inspection and "un-messing", and offered to take a look via Teamviewer, if I buy their software "Recover Explorer Professional" (which seems to be a somewhat trimmed down version of UFS Explorer). I wasn't quite comfortable with the thought, since there was some sensible data affected.
But I downloaded the demo version of the software and used it to scan the drive for lost APFS partitions. It did indeed find partition data of the Downloads partition and allowed me to scan it for files after entering the encryption key. After hours of scanning, it showed me all of the files in a "virtual file system" (including previews), which looked pretty complete. The demo version can only recover files with a limited size (I think 768kb or so), so I bought a license (182€) and a new drive, and I'm now in the process of recovering the files to the new drive. The process is pretty slow (the first 40GB took around half an hour via USB 3.0), but I'm happy that it's working. It looks like I'll get all the data back, and I guess the price is OK for not having a backup...

 

[Taipan]

A question just out of curiosity:
If the partition table and encryption key information on the drive are lost and recovery software thus cannot find a file system, would it make sense to repartition and format the drive with the same file system, same partition size and same encryption passphrase and then try a recovery process? The recovery software would then have a partition table and encryption data matching the original to work with. Or would the encryption key still differ from the original one, despite the same passphrase being used?

 

[DeltaMac]

So, did your try at recovery work?

I have always heard that, if you want to have any chance at recovering "lost" data on a volume/drive, then you should try recovery without modifying the drive. By that I mean that you should not try "random things" that don't help with recovery of the data. Each erase would make it less likely to recover anything, and any recovery method would rely on knowing where the data originally starts. If you reformat (new partition), particularly on a spinning hard drive (I am guessing that those 6 to 12 TB drives are NOT SSDs ) then the "lines" where all the data is written will be recreated, with a new "start point", making it not just unlikely, but impossible to recover anything. The additional challenge of that data being encrypted already, really means that you now have no reason to suspect that you will get any of that encrypted data back.
Have you contacted anyone who can help you? try Drivesavers. They will give you a quote. Not gonna be cheap - but they know what they are doing. Only you can decide if the data is worth the cost.

 

[Fishrrman]

Encrypted?
APFS?
Multiple drives?

(sigh)
My thoughts:
You're not getting that data back.

As Delta suggested, try an outfit like Drivesavers.
But... be prepared to pay multiple thousands of dollars, IF they can get it back.

This is why I NEVER use encryption of ANY kind on all my drives (except one partition on a drive that is kept in my car).
Nor do I use APFS.
HFS+ and "in the open" all-the-way for me.
I WANT my data to be "easy to get at" in a moment of need...

 

[Taipan]

I was able to recover about 80% or so using Recovery Explorer Professional. The data is not important enough to warrant consulting a professional recovery service.

@DeltaMac: I have tried what I can without writing anything to the drive, of course. So anything I'm trying now is more or less just out of curiosity.
The incident that lead to the loss of the data wrote very little to the drive. So I'm confident that the actual data is still there. To my understanding, what is missing is only the metadata about the partition and file system headers / directory, like the partition table and information about the APFS encryption (sorry, I don't know the correct technical terms).
So if these can be recreated by repartitioning the drive in the exact same way as it was partitioned before, it might leave me in a state that's comparable to a working file system with merely deleted files. Meaning that all the file data as such is still there, it's just the file system's directory that keeps track of which file is written in which sectors that's missing. Just like when you delete files on a working file system. Those are usually 100% recoverable as long as you don't overwrite the actual space, because it's just the directory entry that gets deleted.

Edit: Doesn't work, apparently.

 

[Fishrrman]

If you got the data back (even though you may have lost file names and folder hierarchies), then consider yourself lucky.

If you have a lot of "unnamed" files, the solution may be to go through them "manually" and re-name them, etc.

 

[Taipan]

I'm starting to doubt that it was my fault in the first place. Today another drive's partition table was damaged, and it couldn't be mounted anymore.This is starting to p*ss me off. The DAS is going back.