How to copy A LOT of small files to USB

[Go Amtrak]

I have a Late 2018 MacBook Pro. I'm using BigSur, 11.0.1

I'm trying to copy A LOT (maybe close to 200,000) very small files to a 64 GB USB-C Thumb Drive. It is no more than 25GB worth of files, so the files will definitely fit on the disk, but it is taking a long time. I'm assuming that there is a minimum time it takes to copy one file. Is there a way to speed this up?

I trioed zipping the files and copying them and that worked fine, but then unzipping them on the USB-C drive took forever.

 

[velocityg4]

I’d just use a program called freefilesync. I couldn’t say it’s any faster. But you may set it to ignore errors. Then leave it to run until completed. Rather than finding it’s stopped at 10% in because of an error.

If you stop it or it’s interrupted. You can run the sync again where you left off. If there are any file problems. You may compare again to find the problem files.

 

[Go Amtrak]

I’d just use a program called freefilesync. I couldn’t say it’s any faster. But you may set it to ignore errors. Then leave it to run until completed. Rather than finding it’s stopped at 10% in because of an error.

If you stop it or it’s interrupted. You can run the sync again where you left off. If there are any file problems. You may compare again to find the problem files.

OK. I'll definitely give it a try and will report back.

 

[flowrider]

Or just hold the shift key, select all the files, then drag them all across to the thumb drive. You can also use the "Select All" command.

Lou

 

[Go Amtrak]

Yes. That’s not the issue. The issue is it’s taking FOREVER. Even though it isn’t a lot of data, the extreme number of files seems to be slowing things down. I was wondering if there is a quicker way of doing this?

 

[Go Amtrak]

OK, FreeFileSync doesn’t seem to be going much faster. I can see the advantages of it, when you want to merge two locations, but when used just to duplicate, it doesn’t do much.

Thanks for the tip about Freefilesync. I’m definitely going to hang onto it. 😀😀

 

[joevt]

What if you make a disk image of the files and use Restore in Disk Utilities? I haven't tested this method myself - it might be faster if it does a block copy instead of a file copy. To use Restore, select the destination USB drive's partition, click the Restore button, then select the source disk image. Read the messages carefully to make sure the source and destination are correct.

Are you using HFS+? For Restore, the format of the disk image may need to match what you want for the destination partition. I don't know if it works with FAT partitions.

 

[Go Amtrak]

It’s funny you mention the disk format, because I just had a discussion about this with a friend 1 minute ago. Since the files will mostly be read on a windows machine, I bought a copy of TuxuraNSTF and formatted the drive to that. After talking to my friend, I reformatted it it ExFat and while it’s still sluggish, it’s better than it was before.

The disk image thought is worth a try. I tried doing it with Zip files but that didn‘t work. The copy was light speed, but unzipping on the USB drive was anything but.

 

[joevt]

It’s funny you mention the disk format, because I just had a discussion about this with a friend 1 minute ago. Since the files will mostly be read on a windows machine, I bought a copy of TuxuraNSTF and formatted the drive to that. After talking to my friend, I reformatted it it ExFat and while it’s still sluggish, it’s better than it was before.

The disk image thought is worth a try. I tried doing it with Zip files but that didn‘t work. The copy was light speed, but unzipping on the USB drive was anything but.

Was the zip file fast because it was compressed (what percent?) or because it's a simple sequential write? For my idea with the restore disk image, I am hoping that it will do sequential write (by block) and not individual files. If you want to ensure simple block copy, then dd is the way to go (or ddrescue).

 

[velocityg4]

OK, FreeFileSync doesn’t seem to be going much faster. I can see the advantages of it, when you want to merge two locations, but when used just to duplicate, it doesn’t do much.

Thanks for the tip about Freefilesync. I’m definitely going to hang onto it. 😀😀

It can also do a one way sync (mirror). It's actually a very useful program. That can do a lot of complex syncs.

Why not just leave it running? It probably would have been done by now. You can just minimize and go about your day on the computer.

When it comes down to it. Most USB Flash drive are horribly slow outside a sudden burst. They slow down in long transfers. With tiny files. They are often limited to ridiculously slow speeds. The only way it'll transfer those small files faster is by getting a faster (4K files R/W speed is what matters) Flash drive.

It can move the Zip file fast. Because you turned it into one big file. It takes forever to unpack. Because you are turning it into (writing) a bunch of tiny files.

If this is a one time transfer. Pack it into the zip file. Then don't unpack it until you've copied it onto the other computer's SSD. Then let that computer unpack it.

If this is going to be a regular thing. Get a faster flash drive. Better yet. Use a portable SSD. A good one because of the high IOPS needed to transfer tiny files quickly. The big speed number doesn't matter as much as the IOPS number for this. Since the best flash drive is still going to suck at transferring a boat load of tiny files. Even SSD aren't that impressive but much, much better.

Also if this is going to be a regular thing but only some of the files are changing. The Freefilesync can be set to a variety of modes for syncing. So you can speed things up by having it only sync changes (new, updated and deleted files). If both these computers are on the same network. You can turn on file sharing. Just doing this directly between computers. Heck there's methods to keep in sync over the internet too.

 

[me55]

...The issue is it’s taking FOREVER. Even though it isn’t a lot of data, the extreme number of files seems to be slowing things down. I was wondering if there is a quicker way of doing this?

That is normal for most flash drives. If you want small file performance you would need to buy a SSD.

I'd just start the copy process and let it run over night...

 

[ignatius345]

I think there's a lot of Finder overhead associated with copying lots of small files, and that's being compounded by the slowness of this being a flash drive. As @me55 suggests, you might just want to give it time.

 

[appltech]

Try
sudo ditto [source] [destination]
command in Terminal.
Type sudo ditto, drag the folder that contains you 200K+ files , drag the folder (drive) you want to copy them to, and press Enter. (there must be a 'space' symbol between all entries)

 

[ewu]

zip and unzip is always best options for lots of small files.

 

[dwig]

...
The disk image thought is worth a try. I tried doing it with Zip files but that didn‘t work. The copy was light speed, but unzipping on the USB drive was anything but.

Is there a need to unZIP the files on the USB drive? A modern Windows machine can "see" the files in a ZIP archive without unZIPping it and can extract single files when desired.

 

[dwig]

I think there's a lot of Finder overhead associated with copying lots of small files, and that's being compounded by the slowness of this being a flash drive. As @me55 suggests, you might just want to give it time.

Yes, it's the nature of file systems that each file needs to be saved to a particular starting location and that location along with the file's end location needs to be stored in a special data file. That data file needs to then be read to determine the location for the next file, ... rinse and repeat as the shampoo bottle says. All of these reads and writes take a lot of time when there are a lot of files. A single file, like a ZIP, only requires a single read and single write. BTW, this special data file is hidden and its info is used by the OS to create displayed Directory. Each folder on the drive will have an additional directory data file.

 

[gnasher729]

I have a Late 2018 MacBook Pro. I'm using BigSur, 11.0.1

I'm trying to copy A LOT (maybe close to 200,000) very small files to a 64 GB USB-C Thumb Drive. It is no more than 25GB worth of files, so the files will definitely fit on the disk, but it is taking a long time. I'm assuming that there is a minimum time it takes to copy one file. Is there a way to speed this up?

I trioed zipping the files and copying them and that worked fine, but then unzipping them on the USB-C drive took forever.

Yes, unzipping them on the thumb drive gets you the exact same problem. Assuming you want the files on another Mac, you'd copy the zip file to the Mac, then unzip on the Mac.

 

[jdw13]

Being an older guy, I was going to suggest tar from CLI, (/usr/bin/tar) but zip should serve just as well.

 

[Slartibart]

Being an older guy, I was going to suggest tar from CLI

CLI... AmigaOS... aah, the memories 😏😁

 

[posguy99]

A USB2 stick or a USB3 stick?

 

[Slartibart]

The venn diagram for USB-C sticks of Good-Fast-Cheap overlaps in “Does not exist” . Independent of the OS you’re using them with file number, file size, thermal throttling, write cache buffer, optional hub, “marketing speech specs” etc. will have undesired impacts on copy speeds.

If you regularly have to copy hundreds of thousands of files to an external medium, well, use at least a SSD like a Samsung T5. Even faster, use it with the “native” file system - on a Mac APFS.

If you want to use the external medium between machines or devices with different OS use ExFAT (I am fully aware of the theoretical advantages of NTFS, but the implementation of NTFS support outside MS Windows has quite some variance, EXFAT works out of the box everywhere).

If a Mac is involved exclude the external medium from the Spotlight index via the System Preferences.

 

[BrianBaughn]

If you insist on a USB stick get a fast one…as has been mentioned. If the product description doesn't mention fast write speeds you can be pretty sure it'll be sloooow.

 

[joevt]

If you insist on a USB stick get a fast one…as has been mentioned. If the product description doesn't mention fast write speeds you can be pretty sure it'll be sloooow.

With a Sandisk Extreme 64GB, I get 180/245 write/read.
https://www.newegg.ca/sandisk-model-sdcz80-064g-gam46-64gb/p/0BD-000B-000V6
I got that one 5 years ago. There are newer versions now that are half the price:
https://www.amazon.ca/Sandisk-Extreme-Type-Flash-Drive/dp/B01NARBPI7