Third-Party Incremental Backup Software for MacOS

[watcher00090]

Hi,

Can anyone suggest a good third-party backup program for MacOS which supports fast incremental backup? I'm looking for a program which can operate in a matter so that I don't have to re-backup the entirety of a large database or large binary file when only a single part of it changes, rather, only the part that changed should be backed up.

Thanks,
James Pedersen

 

[Lifeisabeach]

Hi,

Can anyone suggest a good third-party backup program for MacOS which supports fast incremental backup? I'm looking for a program which can operate in a matter so that I don't have to re-backup the entirety of a large database or large binary file when only a single part of it changes, rather, only the part that changed should be backed up.

Thanks,
James Pedersen

Carbon Copy Cloner can do incremental backups.

Carbon Copy Cloner | macOS Backup Software | Bombich Software

bombich.com

 

[kitKAC]

Carbon Copy Cloner can do incremental backups.

Carbon Copy Cloner | macOS Backup Software | Bombich Software

bombich.com

But not in the way that @watcher00090 is asking I think, seems like he wants something that creates deltas of files.

 

[Lifeisabeach]

But not in the way that @watcher00090 is asking I think, seems like he wants something that creates deltas of files.

Ah. I didn't quite catch on to that bit at first.

 

[Bigwaff]

Can anyone suggest a good third-party backup program for MacOS which supports fast incremental backup? I'm looking for a program which can operate in a matter so that I don't have to re-backup the entirety of a large database or large binary file when only a single part of it changes, rather, only the part that changed should be backed up.

Sounds like you are looking for file system block level incremental backups. It's been well over a decade since I've used a product called Retrospect, but I believe it had this feature. Retrospect is still around, supported, and updated for macOS, although I don't know if the block level backup feature is supported any more for macOS.

 

[MacsolverUK]

Hi,

I have been using ChronoSync for many years and it has "saved my bacon" many times. My main CAD Mac Pro 2010 has a 3 NVME blade RAID and I use Chronosync to mirror the RAID every half hour to two spinning HDDS alternately and save the last 5 copies of each changed file in an archive. Useful for when I accidentally delete a CAD element - I can just open up an Archive CAD file and copy/paste the missing element. I have a second Mac Pro as a backup server running ChronoAgent and as soon as it boots up on the network, all the Macs do a daily backup. In addition, I can plug in external drives to the Backup Mac and ChronoSync will auto back all new or changed files (keeping the last 5 copies in an archive) as soon a the disk mounts on the desktop. It is easy to exclude items and folders like .Trashes.

I have bootable Clones too, and I use Chronosync to keep the Documents, Desktop and Downloads folder up to date and archive the last 5 copies.

The Best Mac Backup and Sync Software - ChronoSync | Econ Technologies

ChronoSync is a complete, all-in-one solution for file & folder synchronizations, incremental backups, bootable backups (a.k.a clones) and cloud storage. In use by thousands of photographers, video editors, filmmakers, musicians and other data professionals. No other app can compare!

www.econtechnologies.com

The ChronoSync backups have the same file structure as the original, so if you main Mac will not boot, you can plug a ChronoSync backup into another Mac and carry on until you can fix the original Mac. No software needed to use the backups.

ChronoSync "SmartScan" and "Disect packages" may be able to do what you ask. Try contacting ChronoSync.

 

[gilby101]

ChronoSync "SmartScan" and "Disect packages" may be able to do what you ask

Chronosync is still a whole of file backup.

I'm looking for a program which can operate in a matter so that I don't have to re-backup the entirety of a large database or large binary file when only a single part of it changes, rather, only the part that changed should be backed up.

I doubt you will find what you want.

Can you guarantee that the database/files will be closed whenever you do a backup?

What is the database product? What is the large binary file product? I ask because you may need a backup tailored to those products.

 

[Fishrrman]

I was going to post earlier what gilby posted above in #7.

I doubt any commercial BU app can do what the OP wants.

Question:
HOW LARGE is the database file(s) you want to back up?

 

[watcher00090]

@Bigwaff I downloaded the trial version of Retrospect, and yes, it supports block-level incremental backup, which is nice. I'm using macOS Catalina 10.15.7 .

@MacsolverUK I have reached out to Chronosync support regarding this, and the support representative that I was talking to asserted that Chronosync does not support this.

The database is a 10GB SQLite database with a WAL file and the virtual machine is a VMWare Fusion Workstation virtual machine with a .vmdk disk file. Since multiple processes can read a file at any given time, I don't see an issue here.

 

[gilby101]

Arq might be a solution for you. It does block level backup. Whilst it is aimed primarily at cloud backup, it can be used for local backup too.

But, whilst there may be no issue making a backup can you be sure that there will not be issues on recovery. For your two cases:

VMWare Fusion virtual machine with a .vmdk disk file.

I have never trusted a file level backup of an open VM. This is because the backup is equivalent (at best) to a crash of the VM. There is no guarantee that the file system within the .vmdk is in a consistent state and this is made worse by the potential for changes to the VM's file system between reading the beginning and the end of the .vmdk.

For a backup of a live VM you should use a Fusion snapshot which does guarantee a consistent file system in each snapshot. For backup this can be automated with Vimalin which uses vm snapshots to create a consistent copy of the vm in another folder/disk. I strongly recommend Vimalin (and no I am not the author - it is way beyond my skills).

Even if the VM is not running at each backup, I am not sure of the value of what you suggest as there will have been changes throughout the .vmdk whilst the vm was running. You would need to test how much you gain from incrementals.

The database is a 10GB SQLite database

I will pass on commenting about that except that I would be nervous about any backup of an active database.

 

[ChrisA]

Hi,

Can anyone suggest a good third-party backup program for MacOS which supports fast incremental backup? I'm looking for a program which can operate in a matter so that I don't have to re-backup the entirety of a large database or large binary file when only a single part of it changes, rather, only the part that changed should be backed up.

I think Time Machine already does this if you are using APFS all around. In fact, I am pretty sure it ONLY copies the changes.

Apple's documentation does not say exactly how it works, but it does say that TM looks at changes in the file and copies the only changes.

Of course, it LOOKS like the entire file is copied, but it is not. This is just how APFS works.

APFS is a "copy on write" files system. This means that if you open a file, make a change and write it back, only the original files plus the changes blocks are on the disk. The old blocks are kept, so now macOS can go back a version without even needing a backup system. Try this experiment. Fins a 10 GB file. and make a copy of it into the same folder. The disk should NOT have 10GB less free space. The copy does not take up any space because there are no changes. Only changed data takes up space

Time Machine only has to copy the new data (if you are using APFS.)

 

[gilby101]

Time Machine only has to copy the new data (if you are using APFS.)

Are you sure of this? Any evidence?

My understanding is that copies of files are cloned (take no additional space) on the same disk, but not when copying to another disk. Your experiment is all on the one disk, so is not relevant to TM.

 

[hobowankenobi]

In case anybody is curious: https://kb.vmware.com/s/article/1013628

How about a backup tool running on the VM itself, coupled with scheduled full copies of the VM?

 

[ChrisA]

Are you sure of this? Any evidence?

My understanding is that copies of files are cloned (take no additional space) on the same disk, but not when copying to another disk. Your experiment is all on the one disk, so is not relevant to TM.

The first time the file is copied by TM, yes the entire file has to be copied. Bt the second time the file is already on the TM disk. I thnk this does what the OP wants. A full copy only once and there after only changes.

That said, the best way is use the native facility of the VM and database.

A VM will run an OS and you can use that OS's backup utility and the database will have some way to keep a remote mirr up to date by sending only the changes.

 

[FrodeLarsson]

Time Machine (Built-in): While not third-party, Time Machine is a built-in macOS feature that provides incremental backups. It's user-friendly and great for most users' needs.

Carbon Copy Cloner: This software allows you to create bootable backups and perform incremental backups to an external drive. It offers advanced customization options.

SuperDuper!: Similar to Carbon Copy Cloner, SuperDuper! provides features like bootable backups and scheduled incremental backups.

ChronoSync: This software offers synchronization and backup options, allowing you to schedule incremental backups and keep files synchronized across different devices.

Retrospect: A comprehensive backup solution that supports incremental backups and provides advanced features like data deduplication and cloud integration.

Acronis True Image: Known for its cross-platform compatibility, Acronis True Image offers incremental backups, disk cloning, and cloud storage integration.

GoodSync: GoodSync provides both synchronization and backup capabilities, allowing for scheduled incremental backups between devices and cloud storage.

Backblaze: While primarily a cloud backup service, Backblaze offers B2 Cloud Storage, which can be used for incremental backups using third-party tools.

Arq Backup: Arq supports incremental backups to various cloud storage providers, ensuring your data is securely stored offsite.

Duplicacy: Duplicacy is a versatile backup tool that supports incremental backups to local drives, network storage, and cloud storage services.

Restic: An open-source backup solution that supports encryption and deduplication, allowing for efficient and secure incremental backups.

Syncovery: This software offers both synchronization and backup capabilities, allowing you to perform incremental backups and keep files up-to-date.

 

[gilby101]

Time Machine......

A good summary of Mac backup apps.

But, there are two types of incremental.
1) Where the increment is at the file level. That is, only changed files are copied to the backup store.
2) Where the increment is at the "block" level. That is, only changes within changed files (not the whole files) are copied.

The OP is interested in the 2nd type of increment. Do you have a clear idea how each of your backups performs in this respect?

My understanding:
TM, CCC and SD I believe are file level incremental, but would be pleased if someone has evidence to the contrary.

Chronsync and GoodSync do file level increments. (Based on my experience)

Arq Backup and Duplicacy do block level increments (and deduplicate). Again, based on my experience. My understanding is that Restic and Syncovery are the same (though I don't have any experience of them).

Retrospect and Acronis I have no knowledge of their details.

I don't think any of the above are suitable for active VMs or large open databases on a Mac.

 

[ChrisA]

A good summary of Mac backup apps.

But, there are two types of incremental.
1) Where the increment is at the file level. That is, only changed files are copied to the backup store.
2) Where the increment is at the "block" level. That is, only changes within changed files (not the whole files) are copied.

The OP is interested in the 2nd type of increment. Do you have a clear idea how each of your backups performs in this respect?

My understanding:
TM, CCC and SD I believe are file level incremental, but would be pleased if someone has evidence to the contrary.

I think TM is block level if you are using APFS.

 

[watcher00090]

@gilby101 Thank you for mentioning Vimalin, it might be useful to me later.

@hobowankenobi Thanks for posting this, it's good information.

@ChrisA I will most likely not be using APFS on my backup hard drive because APFS seems to be lacking documentation and also because I think the journaling mechanism of HFS+ is better than the copy-on-write mechanism of APFS.

 

[ChrisA]

If you are worried about security, APFS is the way to go.

If for no other reasons...

• APFS uses checksums that protect both data and metadata. HFS can have silent data corruption that is not detected until you actually load the data into the app and see that it is wrong. Metadata corruption in HPS can only be detected by actively scanning by FSCK. APFS is in effect checked every time it is read.
• APFS can have checkpoints and can be rolled back to the last known good point.

APFS is based on ideas from Sun Microsystem's "ZFS" but was simplified for use on smaller personal computers.

 

[hobowankenobi]

While no TM expert (been using other things—including much listed above) I second the notion that I trust APFS much more than HFS+ overall. One of the main reasons I never fully trusted TM was because of HFS+ limitations.

Having said that, I would also test the heck out of TM via APFS regarding stability, recovery, and well as performance (both speed and space used) when working with VM copies or snapshots.

Retrospect used to be rock solid...but that was 20+ years ago (including back on OS 9 and 8!), and they have changed owners a couple of times since then. No idea about them now, nor how it would handle VM challenges. This looks helpful.

 

[hobowankenobi]

I would add a couple more to the great backup list above (though not likely stand-out choices for this VM question):

Get Backup Pro: Not sure what the actual backup engine is...pretty sure it is not Rsync. It has worked for me a few times where others have not, including digging data out of a failed TM backup.

Smart Backup: Great little interface with a custom sync engine, works with zero issues to a mounted NAS share, very good performance. Free!

 

[watcher00090]

@ChrisA I'm not super worried about security, though.

Even though I agree that HFS+ does not offer data checksums out of the box, the man page for tmutil on my computer asserts that:

Beginning on OS X 10.11, Time Machine records checksums of files copied into snapshots. Checksums are not retroactively computed for files that were copied by earlier releases of OS X.

@ChrisA See this apple.stackexchange.com post for more information about how Time Machine checksums work.

 

[gilby101]

APFS uses checksums that protect both data and metadata

For clarity, APFS checksums protect the file system structure, not data and metadata of files.

 

[watcher00090]

I ended up buying a Retrospect license. I like that it supports block-level incremental backup.

In addition I'm also planning to use Time Machine to back up the laptop. I may try to split up the large .vmdk hard disk file into multiple smaller files.

Does anyone have any insight regarding SQLite database configuration which would make it so that the size of the Time Machine incremental for the 10GB database file is not 10GB every time I run a Time Machine backup of the laptop?

James Pedersen

 

[watcher00090]

Actually, I just moved the 10GB SQLite database off the mac laptop and onto to my Windows PC, so the large SQLite database taking up too much space in my Time Machine backups isn't an issue for me any more.