Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.

jtara

macrumors 68020
Original poster
Mar 23, 2009
2,008
536
Suggestions for best free and open source backup solution for MacOS for mid-2021, Big Sur? It's fine if there is a support agreement and/or cloud storage available. But really would like open source and ability to build myself. For obvious security-related reasons.

I am a software developer, mostly iOS/Android, that might have some influence.

  1. Windows, Linux desirable but not necessary
  2. Local storage and S3 destinations, must support non-Amazon destinations (I have Wasabi and IBM Cloud)
  3. Must take and use a snapshot so backup is possible from a running system
  4. De-duplication
  5. Incremental backup
  6. Able to go back in time
  7. Great if it can do something intelligent with database files, specifically PostgreSQL, but not essential
  8. I have concerns about "state" files, e.g. /Library ~/Library, etc. despite my being a dev, I don't have much insight as to what is important to backup there, and what to restore in case of need to do a total restore, so some "guidance" in the app would be great.
  9. Ability to make a bootable backup, or at least a full backup that might not be bootable (I guess there are issues with Big Sur...) would be great but not necessary. I've used Carbon Copy Cloner in the past (yes, not open source...) but one app for both functions would be great. For this, I plug a big hard drive into a USB "toaster".
 

gilby101

macrumors 68030
Mar 17, 2010
2,921
1,616
Tasmania
Some comments on your requirements:

Requirements 2 and 9 will force you into two solutions or, at east, one with two different modes of operation. Bootable backup is not compatible with cloud storage.

Requirement 9: Apple are pushing us away from install recovery (and continue working) provided by bootable backup to one where we reinstall macOS and recover everything else from backup. This strengthens the case for having a standby Mac if the additional restoration time is too expensive from a business point of view.

Requirement 8: You need to be clear about disaster scenarios and what recovery requirements you have. In my case I have different requirements for recovery from local backups (I expect complete recovery to a replacement Mac), but for cloud backup I just need to recover documents, photos, etc. and accept the need to reinstallation of Applications and settings. So little need for backup of ~/Library to my cloud backup destination, but essential for local backup.

Requirement 7: Requirement 3 (which I consider important) is a step towards providing requirement 7. Further than that you should be looking at backup of your databases with a database specific product which can then if required be included in your backup operations. The same thing applies to virtual machines.

Requirement 10: Encryption of backup is essential.

For local backup, you don't need to get anything, Apple provides Time Machine (free) which (particularly with BS's backup to APFS) is reliable, fast and provides for all your requirements except 1 and cloud destinations. You already trust Apple so security reasons for open source are not relevant with this backup product.
Suggestions for best free and open source backup solution for MacOS for mid-2021, Big Sur? It's fine if there is a support agreement and/or cloud storage available. But really would like open source and ability to build myself. For obvious security-related reasons.
In my view the security related reasons don't hold water. You are already trusting Apple (and probably other software providers), so why not a backup vendor? Does building software yourself really make it more secure? Assuming you do have the required skills, do you have the time to go over every line in the code to discover vulnerabilities.

If you really want open source (because you can build it yourself) then your should be using an open source operating system, not macOS or Windows. And do you build your own Linux from source?

Further, requiring both free and open source you are restricting yourself to software (I won't say products) which are likely to be cumbersome to use and, quite likely, unreliable.

Free: I assume you will be paying for your cloud storage, so your solution will never be free.
 
  • Like
Reactions: V.K.

jtara

macrumors 68020
Original poster
Mar 23, 2009
2,008
536
Apple are pushing us away from install recovery (and continue working) provided by bootable backup to one where we reinstall macOS and recover everything else from backup.
The only scenario I care about for bootable backup is in case of drive failure I have something critical and need access to system and data, and thus would boot off of external media (ATA hard drive in USB toaster) in order to get some task done.

Since my primary machine is an iMac, the fix would have to be getting it to Apple for repair and a wait of at least several days. Assuming only the SSD failed, I might be able to get quick access by booting from USB or else doing so on a replacement iMac.

I might not want to take hours (or even an hour) to install the OS before being up and running, if only at a performance penalty and temporarily.

you should be looking at backup of your databases with a database specific product which can then if required be included in your backup operations. The same thing applies to virtual machines.
You're right. Each DB has tools available to take a snapshot using the DBs own snapshot system, so write that to the filesystem first. So, I (ideally) need a backup solution with before/after hooks.

Fortunately, I wouldn't normally have any very large database. It would only be some test data set.

Does building software yourself really make it more secure? Assuming you do have the required skills, do you have the time to go over every line in the code to discover vulnerabilities.

If you really want open source (because you can build it yourself) then your should be using an open source operating system, not macOS or Windows. And do you build your own Linux from source?
It insures that it corresponds as built to the source I might be looking at. (Since I trust the compiler and other build tools...)

"reproducible build" is I realize thorny, and something that is in various stages of development by several crypto wallet, messaging (Threema, Signal), and even Covid exposure (Dutch one seems to have achieved it?) apps.

Git can validate that source is the same for multiple builders, at least.

No, I'm not going to examine every line of code. But if the app is popular enough, perhaps security researchers have poked and prodded and commented publicly.

No, I don't build my own Linux from source. But I like that I could, and that others have. I do build the mobile app platform I use to write apps from source. I suppose more for debugging. I'm very familiar with it's innards, I do sometimes trace into the platform code or even modify to add my own logging or contribute back some fix or feature. I will also look at source when documentation is unclear. I need to know the source I am looking at corresponds to what I am running.

I realize some pooh-pooh the notion that open source enhances security through the potential for open examination. I'm on the side of the fence without the poo. ;)

Main reason I think is that it's possible to fix things myself. In a backup system, that just might come in VERY handy! If I have enough motivation, and there is a bug that prevents me from retrieving data that I need, I have the power to fix it.

Yes, ARQ has an open data standard, and there IS an open-source restorer, so there's that.

requiring both free and open source you are restricting yourself
Open source but not free is OK. Hard to imagine an acceptible enforcement mechanism, other than threat of legal action. Any technical enforcement solution puts control over ability to use in somebody else's hands.
 

gilby101

macrumors 68030
Mar 17, 2010
2,921
1,616
Tasmania
Since my primary machine is an iMac, the fix would have to be getting it to Apple for repair and a wait of at least several days. Assuming only the SSD failed, I might be able to get quick access by booting from USB or else doing so on a replacement iMac.
Some bad news: The new M1 Macs won't boot without a working SSD on the motherboard. But can now (macOS 11.4) boot fairly reliably from USB - useful if the fault is just data corruption. With an Intel iMac there is, of course, not that problem.
Needing fast recovery from M1 Mac hardware failures makes the Mac Mini attractive as it is easy and quick to replace without the cost of a replacement Mac with attached screen.
But I am sure you have worked out what you would do in different fault scenarios.

Yes, ARQ has an open data standard, and there IS an open-source restorer, so there's that.
More bad news:
Though I consider Arq to be the best of the Mac cloud backup tools:
1. I am not convinced that the description of the data for Arq 7 is sufficient to write a restorer - you would be able to tell better than me.
2. The open-source restorer is for Arq 5 and only for AWS and local storage . An indication of the developer's good intentions though not usable for the current Arq 7. Arq 4 and 5 were similar in storage structures. Arq 6 and 7 are significantly different. In case you don't have the link: Arq restorer

I realize some pooh-pooh the notion that open source enhances security through the potential for open examination. I'm on the side of the fence without the poo. ;)
Poo from my side of the fence (Tasmanian wombat).
 

gilby101

macrumors 68030
Mar 17, 2010
2,921
1,616
Tasmania

With Arq, the developer Stefan is very dedicated (and capable), but whilst the spirit is willing there is just too much for one developer to do. For him, I suspect it was a big change to even have someone else dealing with support issues.

To my mind this does highlight the biggest issue with small (or single person) developer teams and closed source software - key person dependency. This is also true in open source (e.g. Duplicacy), but the open source nature makes it easier for others to step up if required.
 
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.