Cloning hard drives - why 3rd party app instead of Apple's Disk Utility?

[OldCorpse]

I have a new M1 MBP (8GB RAM / 256GB SSD) and I'm on BS 11.1.

I have a very large music collection on an external drive - 1.82TB, and that includes the Music library. I have managed to get it to work on my MBP. It has previously been used by me on my late 2009 iMac and High Sierra.

Now, I want to back up this drive and actually transfer everything over to a new SSD I bought the 2TB SanDisk Extreme Portable.

So my idea was to clone my old HDD to the new SSD so that if needed I have a backup instead of just relying on an old spinning drive.

Here was my plan - since the source HDD is on HFS+ Journalled, I'd formate the SSD to also the same HFS+ Journalled. Then I was going to use Disk Utility "Restore" function on my MBP to clone the HDD to the SSD.

There's one additional wrinkle on this, is I want to password protect the SSD, so my idea was to simply create a password protected dmg on the SSD and clone the HDD to that dmg.

Anyhow, I have an excellent app from SuperDuper! which I've used for years to clone and back up my HDDs, but I cannot use it for this project, because apparently SD! does not work with Big Sur as yet, nor does CCC fully.

So I simply thought - why not just use the Disk Utility under Big Sur and accomplish this - all I need to do is clone one drive to the other, and it's not a startup disk or anything. The only thing is that it has to work with the Music App, but I figure if the HDD works now, then cloning it to the SSD should have the SSD mone work with the Music app just as well.

There's a thread on SuperDuper! and CCC for Big Sur, and folks are very sad that it has not been updated yet, but I wonder if I really need anything other than Disk Utility on my MBP?

So my question is - will my plan work to clone from the HDD to the SSD using Disk Utility?

Thank you in advance!

 

[Apple_Robert]

I haven't tried that route before. I let Time Machine backup to two different drives (both encrypted password protected). The first drive is wireless Time Capsule and the second drive I plug in when I want to for a quick backup. Both take just a few minutes. From what I can see, I think your plan will work although it is more involved that the TM route.

 

[OldCorpse]

Thanks, Apple_Robert, I guess I'm not clear on how Time Capsule (i.e. Time Machine) works. My understanding was that TM only backs up the data, but does not preserve the structure of an external drive. So - I thought - you can't simply hook up an external drive to the TM drive and copy over the music to the new drive.

As you can see, because I have so much music (1.82TB), I *must* keep it on an external drive - my MBP (which has only 256GB internal SSD storage) is too small. So now the idea is to clone the external HDD to an external SSD.

Of course, I might be missing something, which is why I've reached out to these boards

 

[Monotremata]

Yeah its not a copy of your actual drive or anything, its just a copy of your files. The root folder is the name of the computer, and then each drive/partition has its own folder under that and your file structure is intact there, just like it appears in the Finder. You can copy files from it all day long if you need to but that won't give you a bootable copy or anything. A TM drive IS bootable, but its booting from a hidden copy of the recovery partition, you have to OPTION boot to even select it.

Disk Utility should work fine. It can sure make and restore an image file to a new drive and run with it, dont see why it cant copy one straight over.

 

[Honza1]

This whole plan looks like major waste of effort and disaster in making. I do not understand why you want to make it this complicated. The music is inside a folder. It may look like file (may be...) but it is folder and it is something you can simply copy. So here is much easier method:
1. Connect new drive to BigSur computer and format as APFS, use encryption of you want it encrypted.
2. When that is done, connect the old drive with the music to BigSur computer
3. Drag and drop the folder with music from old disk onto the new disk.
There is nothing more to it. It will take serious time (spinning drive is limited to at most 100MB/seconds and it likely slower).
Now, if you have TimeMachine backup, it is also easy. TimeMachine disks retain folder structure (facing user in FInder) of the sources, so simply find the folder with music and drag and drop on new disk. Expect much longer time, TimeMachine disks have internally lot more complicated structure and reading them is slower.
Either way, do not bother with this duplicating.

 

[OldCorpse]

Thanks, guys!

Thank you Honza1 for the warning! I just thought that simply copying files over from one drive to the other would lose the organization and metadata so that when I hook up the new drive (with the copied files), it will just be a bunch of music files that would not be recognized by the new Music App on my MBP and would have to be somehow "organized" with all the pointers in place. As you can tell, I have a very hazy understanding of what's required and how it all works, so thank you for letting me know, I truly appreciate it

 

[danmart]

About a year ago (so the details are a little hazy...) I bought a new RAID which I use to hold my media library which comes in at around 1.2 TB so similar scenario to the OP’s.

Moving the data from the old RAID to the new one was pretty easy. Just copy the file structure from the old drive to the new one. There are meta data files in there which tell your Music app what is available, but as long as you move all of them you will be fine. Hold down the Option key when you open the Music app and it will offer you to pick a library; navigate to the top of the new structure and it should see the meta data it needs. That should be job-done.

Just make sure your set-up is rock-solid and won’t shut itself down at any point; the worst scenario here is only part of your files get copied over and you need to try to re-initiate the copy process with a partially copied structure. It will take hours to complete the copy.

 

[OldCorpse]

Well, here's my experience. I first tried my method using Disk Utility "restore" - and that didn't take long, because "Operation Failed" and no explanation, all I could do is press "Done". I tried various ways, but it always failed within 3 seconds or so.

So, feeling like a savage, like a lowbrow peasant, like the lowest level of human being with zero intellect, I selected the HDD icon on the desktop, right-clicked and pressed "Copy" - and then, with utter shame and feeling suicidal from deep embarrasment, clicked "Paste item" onto my newly created 1.9TB dmg. After a bit of fussing (had to input admin password to copy from the HDD), the copying commensed. It has copied 50GB so far in less than an hour and tells me it'll take "about a day" to finish.

I feel like I've entered a salon to play the piano, and while others play with dexterity, selecting "restore" and cloning disk images, I brought a tree branch with which I'm smashing the piano to get sounds out of it, with a brute "copy" "paste" - the whole room is appalled and disgusted, and when finished I'll have to slink into the darkness never to speak of this again. That's assuming that the copy/paste will even work ultimately and I'll be able to play my music from the newly copied to SSD. We'll see. I feel defeated.

 

[IowaLynn]

You still want to shop for a larger drive so it’ll have sufficient free space, I am for 30%

CCC will insure that all files copied are error free.
When copying, you need to insure there is room for temporary files copied along with new file. Your DMG is likely seen as a single file I’m guessing.

SoftRAID 6 might be worth investment - a mirror set - along with two 3 or 4TB drives. Not just 2TB, not when dealing with very close to that on your current media source drive.

 

[Honza1]

Well, here's my experience. I first tried my method using Disk Utility "restore" - and that didn't take long, because "Operation Failed" and no explanation, all I could do is press "Done". I tried various ways, but it always failed within 3 seconds or so.

So, feeling like a savage, like a lowbrow peasant, like the lowest level of human being with zero intellect, I selected the HDD icon on the desktop, right-clicked and pressed "Copy" - and then, with utter shame and feeling suicidal from deep embarrasment, clicked "Paste item" onto my newly created 1.9TB dmg. After a bit of fussing (had to input admin password to copy from the HDD), the copying commensed. It has copied 50GB so far in less than an hour and tells me it'll take "about a day" to finish.

I feel like I've entered a salon to play the piano, and while others play with dexterity, selecting "restore" and cloning disk images, I brought a tree branch with which I'm smashing the piano to get sounds out of it, with a brute "copy" "paste" - the whole room is appalled and disgusted, and when finished I'll have to slink into the darkness never to speak of this again. That's assuming that the copy/paste will even work ultimately and I'll be able to play my music from the newly copied to SSD. We'll see. I feel defeated.

There is infinite number of ways you could have achieved the "complicated" job of copyingy few files from disk A to disk B ;-)
Some elegant, some less. I actually like the fact that the standard method (copy/paste) simply works, as it looks like. It is satisfying, that you do not need anything special and unique. On the other hand, to use your piano analogy... In this specific case you came in the saloon to play with one finger just one note, this is no high science what you want to achieve. So it is correct, that by hitting it with branch you may achieved the same result.
Personally, I would open application Forklift (it shows two disks at once) and drag/drop the folder there. It is bit smarter on how to manage copy of large number of files and allows you to pause if needed. But, at the end, it just tells Finder to copy each file so it's doing what you do.
You could use CarbonCopyCloner, which would be very elegant and do this efficiently and smart. And could check for errors, as pointed above. But you would need to download it and learn a new interface. Would that be worth your time? Sometimes just get it done and go on with your life is best.
It is taking long time since music files are often quite small and copying many small files takes lot longer than one large one. Or, the source disk has reading issues and system is challenged to read from it.
Verify your new data before you do anything with the old drive.

 

[haralds]

I think as a Mac User it is a crazy not to have Carbon Copy Cloner or ChronoSync. For the price of a few cups of coffee you get convenience, security, and performance.
I have both. They each have a different emphasis.

 

[maverick100]

CCC!!! I have been using them for years; For me they have been just the best.

My System 2-Thunderbolt 3, 4-bay Akitio Quad X enclosures; 12tb drives;
2-Thunderbolt 2 Akitio enclosures; 8tb and 6 tb drives.

Master and backup onsite; third backup offsite in safety deposit box.

Itunes library 10tb; 90% is video.

backups run daily.

Two backups of 2017 iMac; on site, on Mojave (will not be moving to Catalina or Big Sur)

CCC has the best Support ever. I have asked them questions not related CCC and they have been willing to share.

My only question how important is your data. For me the cost of quality drives and software doesn't matter.
Speed of backups; secondary to quality of Backus.

Just did my monthly backup of iTunes library; I use the option to look for corrupted files when doing this backup; It took 35 hours to complete.

Best backup software ever.

 

[mikzn]

I'll add my vote for CCC - use it to back up music and other media (Photos libraries and Movies) 3 times a week to separate drives in case of a "bad back up" one of the back ups is to Time Capsule (using CCC)

I use the "incremental" / "No Safety Net" settings and back up is ussually finished in a few minuites to external SSD or 30min to Time Capsule (slower due to HDD and wireless)

 

[Apple_Robert]

There is infinite number of ways you could have achieved the "complicated" job of copyingy few files from disk A to disk B ;-)
Some elegant, some less. I actually like the fact that the standard method (copy/paste) simply works, as it looks like. It is satisfying, that you do not need anything special and unique. On the other hand, to use your piano analogy... In this specific case you came in the saloon to play with one finger just one note, this is no high science what you want to achieve. So it is correct, that by hitting it with branch you may achieved the same result.
Personally, I would open application Forklift (it shows two disks at once) and drag/drop the folder there. It is bit smarter on how to manage copy of large number of files and allows you to pause if needed. But, at the end, it just tells Finder to copy each file so it's doing what you do.
You could use CarbonCopyCloner, which would be very elegant and do this efficiently and smart. And could check for errors, as pointed above. But you would need to download it and learn a new interface. Would that be worth your time? Sometimes just get it done and go on with your life is best.
It is taking long time since music files are often quite small and copying many small files takes lot longer than one large one. Or, the source disk has reading issues and system is challenged to read from it.
Verify your new data before you do anything with the old drive.

Speaking of Forklift... I recently purchased it less than a week ago. I did so after seeing so much praise for it here. And even though I do need to use it every day, the price of the app is well worth it. I really like the dual pane approach. The app makes it very easy to see and access all devices on the network and dragging and dropping is very nice indeed.

 

[IowaLynn]

About a year ago (so the details are a little hazy...) I bought a new RAID which I use to hold my media library which comes in at around 1.2 TB so similar scenario to the OP’s.

Moving the data from the old RAID to the new one was pretty easy. Just copy the file structure from the old drive to the new one. There are meta data files in there which tell your Music app what is available, but as long as you move all of them you will be fine. Hold down the Option key when you open the Music app and it will offer you to pick a library; navigate to the top of the new structure and it should see the meta data it needs. That should be job-done.

Just make sure your set-up is rock-solid and won’t shut itself down at any point; the worst scenario here is only part of your files get copied over and you need to try to re-initiate the copy process with a partially copied structure. It will take hours to complete the copy.

I assume you are referring to having everything on Uninterruptable Power (UPS). No room for power failure, even momentary.

 

[danmart]

I assume you are referring to having everything on Uninterruptable Power (UPS). No room for power failure, even momentary.

The risk wasn’t quite so great as to necessitate that for me. More like: make sure your machine isn’t set to turn itself off over night to save power (I used to have mine set up this way in the past). Things like that.

 

[OldCorpse]

Folks, thank you for the discussion - very illuminating! However, as I have pointed out in my OP, I am well aware of the advantages of apps that allow smooth copying and updating of drives. The problem is not that I don't want to use one. In fact, as I wrote, I do have SuperDuper! and have been using it since before I got my late 2009 iMac, and I love it. I also regularly use it on my other music backup (also an HDD) to keep them in sync. As soon as I add any significant amount of music (or remove any!) to my main music drive, I use SD! to sync it with the backup.

No, the problem is that neither SD! nor CCC work with Big Sur on M1 macs. CCC works on Big Sur on Intel macs, but I have the new M1 MBP.

You're preaching to the choir! I'd have loved to use SD! or CCC (which I was willing to buy, even at $40), but it wouldn't have done me any good. That's why in desperation I resorted to Disk Utility "restore" - which proved to be a complete bust.

And that's how I ended up copy-pasting. I started the C/P process yesterday around 11:30 am PST, and today, it's about 24.5 hours later at 12:10 am PST and it still has 5 hours to go according to the estimate - this is more than 24 hours. If this holds, then copying 1.82TB of music from a 7200rpm HDD with a micro USB3 connection from the HDD through a USB-C on the MBP to a 2TB SanDisk Extreme USB-C to USB-C on the MBP takes apparently 29 hours.

There's one additional wrinkle in all of this. The MBP has only 2 TB connectors. If I plugged in each drive into its own MBP connection, I'd have to rely on battery power exclusively as there is no power pass-through on either of those drives to the MBP. That means I would not be able to do this, because the battery would have collapsed mid-through the process.

So this is what I had to do:

1)Buy a USB hub - I bought the AUKEY USB C Hub 12-in-1 Type C Adapter from Amazon for $56. This is a bare minimum, as it has only one USB-C data input connector, plus another one that only delivers power.

2)Buy a connecting cable for the HDD - UGREEN USB-C to USB 3.0 Micro B Cable for $8 to maximize the data transfer speed from the HDD.

Now, there are pecularities to the hub I purchased - and you won't find that out until you actually buy it, even though I read all the reviews. So I may as well do a mini-review should you be considering buying one.

One, you can't plug in other drives into the AUKEY hub and then into the MBP - they won't be recognized; you have to first plug in the hub to the MBP and then the drives into the hub. Next, you can't plug in the USB-C power on the hub to the electrical outlet while there's any activity on the drives, because it will pause for several seconds and you'll lose data. Instead, you must *first* plug in the power from the outlet to the power USB-C on the hub, that way you don't get any interruption. So my sequence was plug in the hub to one of the USB-C connectors on the MBP ---> then plug in power from the outlet to the USB-C on the hub ---> then plug in the UGREEN cable from the HDD to the data USB-C on the hub. That takes up the only data USB-C outlet on the hub - why not use that outlet for the SSD instead, because the hub USB-C is speed limited to 5Gbps (which you only find out if you read the specs on the manual supplied, and won't read it in the text on Amazon), whereas the SanDisk Extreme SSD is rated for 10Gbps. Of course, ultimately it doesn't matter, because the speed of copying will be rate limited by the HDD, so the 10Gbps from the SSD will be irrelevant. Now, the hub is OK, nothing special, it works, I used the USB 2.0 ports to transfer data from one thumb drive to another and also to the MBP, so it's fine. For the operation I needed, this hub was the bare minimum allowing the charging of the MBP and plugging in the HDD to a USB-C input at the same time.

I then plugged in the SSD directly into the second USB-C connector on the MBP. And now I could commence operations.

The point being, is that if you think that all you need is the MBP to copy one drive to another, you are simply wrong. Depending on the kind of drives you have, you will need hubs and connectors due to the MBP having only two TB connectors and obviously being dependent on power from one of them should the battery be insufficient. Would it be nice if the MBP had 4 TB connectors - yes, but I understand that this is Apple's first M1 and lowest level of MBP with pretty much the same options as the Intel equivalent (also only 2 TB connectors).

Anyhow, as you can see, as of end of December 2020, beginning of January 2021, copying data between drives using the M1 MBP on Big Sur 11.1.0 is a complicated affair and apps such as SD! and CCC are not options at this time, while the Disk Utility under BS 11.1.0 is quite faulty/buggy and limited. It's early days, but it's also clear that there are serious limitations to both the operating system of BS and the M1 macs as far as such basic user needs as copying and transfer of data. Early adopters should be aware of these limitations, which will hopefully be ironed out in the future with updates to the OS and hardware limitations such as number of connectors. YMMV.

I'll give a final update once the copying is finished.

 

[IowaLynn]

You don't need a bootable clone.

macOS Big Sur Known Issues | Bombich Software

bombich.com

I am choosing to wait for - 11.4+ as usual, skip M1 and using Late 2020 Intel 6-core Mini 16/512 until bugs are ironed out - and my 4 ports work.

Ditto has been around for over 50 years and still main tool.

 

[OldCorpse]

I hear you, IowaLynn, and I'd have done the same - rode my late 2009 27" iMac until they had an Apple Silicon iMac. But unfortunately, with rather bad timing, my ancient Seagate HDD in the iMac started making worrisome clicking noises and generally slowing down dramatically. I needed something immediately, and wasn't about to buy an Intel mac. The only one for *immediate* purchase was the MBP 8GB RAM and 256GB SSD. I couldn't even buy - immediately - at least a 512GB SSD. Be that as it may, I figure, this MBP was cheap enough to hold me over until I can buy a proper second generation Apple Silicon iMac and maybe a re-designed Air whatever generation - all in the next couple of years.

On a completely different note - and sorry to throw this in like that - but I have a cat whom I love dearly, but who is a major saboteur that is a master destroyer of electronics through very sneaky and adroit walking on keyboards. He presses key combos with one pass such that it sometimes takes me hours to figure out what he did and how to disentangle the mess he creates. He was very clearly fascinated by the new MBP and spinning elaborate plans on just how to step all over the keyboard and mess up whatever it was I was doing. Since the copying takes hours I couldn't simply leave the MBP open for him to walk all over it. I therefor closed the lid - but I didn't know if the MBP would work in clamshell mode to copy so much data, so I took a small yellow stickies pad and split it in half and put each on both ends of the keyboard so that the lid would not close 100%, just in case clamshell doesn't work. So two points:

1)is it safe to completely close the lid while the copy paste operation is chugging along?

2)I'm giving away for free a multi-million $ idea to any enterprising programmer out there - the problems with cats walking on keyboards is a universal problem the world over. There will be millions upon millions of customers for any app that can disregard cat walking on keyboards. I imagine you could use Machine Learning to figure out that random key-combos from cat feet form specific patterns which could be recognized and rejected by the software. I know I'd pay a pretty penny to install such a defense against my Cat the Saboteur... let him walk all over and accomplish nothing!

 

[OldCorpse]

As promised, I'm updating - it seems the operation was finished successfully. It took approx. 29 hours to copy 1.82TB of music files from a 7200 spinning Seagate HDD to a 1.9TB password protected dmg on a 2TB SSD SanDisk Extreme. The HDD was connected to a hub through a USB 3.0 Micro B - USB-C cable and the SSD was directly connected to the MBP. It was a simple Copy & Paste operation after the "Restore" operation on Disk Utility failed immediately, within seconds of picking source to destination drive.

The good news is that after a lot of additional hardware you can successfully copy and paste a lot of data from one drive onto the next through the M1 MBP 8GB RAM, 256GB SSD on Big Sur 11.1.0. The bad news is that Disk Utility on the MBP is pretty underwhelming and certain funtions on it just don't work, period. Hopefully further refinements in Big Sur and perhaps even hardware upgrades down the road in future Apple Silicon computers will allow ordinary users to accomplish very ordinary things, like copying data and cloning disk images without jumping through inordinate hoops.

Thank you everybody!