Arq

[Gixene]

Hello MacRumors Community,

Note: I would post this thread in Mac Apps and Mac App Store, but I feel as if those members here in the Mac Basics and Help would be of more help in this topic.

Since I do not have an online backup solution, I began searching. I found a provider called Arq. Typically I don't make rash decisions, except in this situation. Arq immediately seemed to be an excellent solution. Suddenly, I began downloading Arq's application due to it being free for 30 days. Mainly, I wanted to see the UI of the application but didn't want to actually backup data until I had more understanding. Then I found Amazon is offering unlimited cloud storage for 3 months on Amazon Drive. One thing lead to another, I entered the Amazon Prime account information and automatically began backing up. Now I have many questions and I need more understanding.

1.) Arq defaulted to the backup location of "user/name" or the technical term 'Home Folder' with the iconic image of a home. However, I did choose "Exclude items skipped by time machine" Is this sufficient for backup of data?

2.) I use 37.7 GB of 256 GB on Macintosh HD. Though Amazon Cloud Drive shows 1.4 GB of storage while the first backup 14.003 GB according to Arq's log. Does this seem normal?

3.) How does Arq operate? If I needed to restore from backups, would I need to install the OS or is the OS apart of the backup or would I need to install OS then restore?

4.) Before viewing the backup on Amazon Cloud Drive, I envisioned that I would be able to view files, pictures, etc. Except, the backup is quite different than I had expected. Is this normal? Is it due to the encryption? Essentially, I'm unable to view a picture or document.

5.) I don't use Time Machine. I'd prefer not explain the reasoning(s). Please, respect the this decision and leave it at that. With that said, I would prefer to use Arq's versioning to mimic Time Machine. I never delete data that should not have been deleted. I also never have the urge to return to a prior version of OS. However, I want to reduce the risk of data corruption. I use Carbon Copy Cloner with Checksum once a month. I intend for Carbon Copy Cloner's Checksum feature to provide an early warning of possible corruption and preventing the copying of corrupted data to other backup solutions. Due to Arq not not using Checksum, I wonder if it would be beneficial to use Arq's versioning? (Note: After 5 hourly backups since using Arq, it seems it's using versioning on it's own.) If so, would Amazon's Cloud Drive unlimited storage option be the best option based on cost bases or Amazon S3 or other options Arq offers? Is it recommended to use Arq to backup hourly, daily or weekly? Also, how long should I keep these versions?

Any other opinions, advice, knowledge I should know?

Thank you, Gixene

 

[Weaselboy]

Fellow Arq user here and I'll try and answer your questions.

1. Like you mentioned, Arq by default only backs up the data in your home (user) folder. This is sufficient to save all your personal information and files. Arq does not backup things like applications or the OS itself.

2. Run the command below in Terminal to show the size of your users folder in GB. That should come close to the 14GB Arq has backed up. The Arq backup will be slightly smaller since it excludes some cache files.

du -hs ~/

3. The OS is not part of the Arq backup. You would first need to reinstall the OS, then you could restore your personal data using Arq. The Arq documentation has a section on how to do that.

https://www.arqbackup.com/documentation/

4. What you are seeing is normal. Arq breaks up your data into encrypted blocks and sends it to the backup server. It cannot be viewed like you can view files on something like Dropbox for example.

5. Like you discovered, Arq does versioning. It also does a monthly file verification similar to checksum. If you really only have around 14GB of user data, Amazon S3 would be much cheaper for you. Heck even if you allowed S3 to store 2X the data you have for versioning, it would only cost you less than one dollar a month vs. five dollars a month for Amazon Cloud Drive. I have about 30GB of user data on S3 with Arq and I set the Arq purge budget at $1.40. So that gives me about 46GB on S3 (140/3 cents a GB=46). That give me plenty of versioning going back a few months. I have Arq set to run once a day. To me hourly is more that I need and IMO just racks up storage space for unnecessary versioning.

Hope this helps. Let me know if you have more questions.

 

[SteveJobzniak]

Arq is totally incredible. And yes it backs up your home folder by default but you can add other folders and drives too. Easily.

- Extremely well architected and efficient. Uses the "commit history" concept from Git to do extremely safe and stable backups, with full revision histories.
- Open file format and open-source tools exist for decrypting backups, so you are never stranded with proprietary backups that can't be recovered if a company goes out of business.
- Supports all major cloud storage providers. So you can choose exactly the storage plan you need.
- Full encryption so that only you can recover the files. Choose a strong password and you're safe!
- You can back up as many computers, users and internal or external hard drives as you want.
- No monthly fees except what your cloud provider charges. This is in contrast to "BackBlaze" and other such services which average $5/month per backed up user for the rest of your life.
- You can search for files/folders and restore them (via the Arq GUI), and can see all historical versions it has backed up of that file/folder.
- Extremely fast and efficient backup daemon, unlike "BackBlaze" and other such garbage services which have bloated, memory-intensive software which slows down your computer.
- Only uploads changes in files, instead of sending the whole file all over again if its contents change. This means it is very bandwidth efficient.
- You can back up external hard disks by connecting them, adding them to Arq, and ticking the 'Skip if volume not mounted' box. The data will stay online forever. This is in contrast to "BackBlaze" and other such services which delete disconnected external drive backups after a few months.
- It does a monthly full verification to check that all backed up data still exists online, and it uploads anything that's missing (which it never is, but it's good to have the feature).
- Arq calculates hashes (aka checksums) for all file chunks it uploads, and only stores identical data once (multiple files with the same content will all point to the same "data" chunk online).
- On Amazon Cloud Drive, it (thanks to my feature request actually) verifies that each chunk was 100% correctly uploaded by checking Amazon's response (which has an MD5 hash) to know that the chunk was properly uploaded and not corrupted in transit. If it was corrupted, it re-uploads that chunk to Amazon.

Personally I use Arq with Amazon Cloud Drive Unlimited, and it truly is "unlimited". Other Arq users have backed up amounts such as 5 terabytes and 10 terabytes without getting complaints from Amazon. And Amazon Cloud Drive uses Amazon S3 as its storage back-end, so it's pretty much as good as S3, but without any storage costs. It has lower backup redundancy than S3 but so what? Arq verifies the data monthly, and Amazon keeps their storage drives in top shape and replaces any hard drive that's about to fail long before it fails. (@Weaselboy: You should really consider switching from S3 to ACD, unless you're using so little data that your S3's storage+transit costs are cheaper than ACD's $4.99/month. At something like 60 GB you've exceeded ACD's monthly cost if you're using S3. I use about 500 gigabytes which would cost about $35 per month via S3, and I will be needing even more when I back up even more computers later, which is why Amazon Cloud Drive Unlimited is perfect for me).

I bought Arq at Version 5, which was a major rewrite where he changed from a per-computer license to a single global license that you can use on all your computers. It's only 1 guy coding it, Stefan Reitshamer, but he's been doing this for years and is extremely talented, thorough and devoted. I foresee myself using this for as long as he keeps making it, and I bet that he'd sell the product to another company so that someone else can continue development if he ever reaches a day where he's too old or sick to carry on. But as of right now, he's pretty young and this is his only source of income and there's no sign of him ever stopping.

Therefore, I bought Arq 5 ($49.99) with the "Lifetime" license (which costs $29.99 extra) for a total of $79.98. That'll keep all of my computers safe for years and is totally worth it. It covers all versions he will ever release. Otherwise you'd pay $24.99 again per each major version upgrade. So the lifetime license has paid for itself by the time Arq 6 is out. And then it just keeps on giving.

Arq is the BEST, fastest, and most powerful online backup solution. Your data, encryption keys and cloud storage solutions all belong to YOU instead of some faceless corporation. I recommend that you buy the lifetime solution and get used to Arq. It's going to serve you well.

As for Time Machine: I use that too, for local network backups with instant restores. To me, Arq is only for the "okay my house just burned down or was burglarized and I have lost all data and decades of work and personal memories" situation. And for that, it gives me TOTAL peace of mind. I'll never lose a project or backup.

As for Carbon Cloner: I never used it. I don't see the point of creating a totally raw image of a disk, which is an image that take ages to create and quickly grows out of date. That's masochistic. I'd rather pick up the data on that disk via Time Machine and Arq. And if something gets screwed up, I do a full OS restore/reinstall via Time Machine, or I reinstall the entire OS on a new machine and restore particular files via Time Machine or Arq. Both of those solutions let me choose the exact point in time I want to restore from.

Time Machine + Arq is a great combo. But I could even survive with only Arq, because I don't really need to recover files with Time Machine very often, and I *could* therefore live with the extra time it would take to download the backed up files from the cloud.

I've actually been thinking of turning off Time Machine and only using Arq, but I have an AirPort Time Capsule 2TB so there's no point turning it off. And it's nice to have the option to do a full OS reinstall via Time Machine, to get back on track quickly if I ever have a normal disk crash.

In short: Ditch Carbon Copy Cloner. Start using Time Machine (backing up to the disk you used to use for your carbon clones) and Arq instead. Because Time Machine does everything Carbon Copy did but much faster, more efficiently and with more granularity and automatic updating of the backup so that it's never outdated. And you can also encrypt your Time Machine backup with a password (so that it stores everything using AES encryption), which is much safer than Carbon Copy Cloner!

 

[Weaselboy]

(@Weaselboy: You should really consider switching from S3 to ACD, unless you're using so little data that your S3's storage+transit costs are cheaper than ACD's $4.99/month. At something like 60 GB you've exceeded ACD's monthly cost if you're using S3.

S3 is much cheaper for me. My bill is about $1.40 pr month. Even at 60GB, S3 would only be $1.80 per month.

 

[chown33]

As for Carbon Cloner: I never used it. I don't see the point of creating a totally raw image of a disk, which is an image that take ages to create and quickly grows out of date. That's masochistic. I'd rather pick up the data on that disk via Time Machine and Arq. And if something gets screwed up, I do a full OS restore/reinstall via Time Machine, or I reinstall the entire OS on a new machine and restore particular files via Time Machine or Arq. Both of those solutions let me choose the exact point in time I want to restore from.

Time Machine + Arq is a great combo. But I could even survive with only Arq, because I don't really need to recover files with Time Machine very often, and I *could* therefore live with the extra time it would take to download the backed up files from the cloud.

I've actually been thinking of turning off Time Machine and only using Arq, but I have an AirPort Time Capsule 2TB so there's no point turning it off. And it's nice to have the option to do a full OS reinstall via Time Machine.

In short: Ditch Carbon Copy Cloner. Start using Time Machine (backing up to the disk you used to use for your carbon clones) and Arq instead. Because Time Machine does everything Carbon Copy did but much faster, more efficiently and with more granularity and automatic updating of the backup so that it's never outdated. And you can also encrypt your Time Machine backup with a password (so that it stores everything using AES encryption), which is much safer than Carbon Copy Cloner!

I just wanted to correct a few errors here.

Carbon Copy Cloner doesn't "create a totally raw image of a disk". CCC uses the 'rsync' command, which means it copies files, permissions, attributes, etc. using normal file-system functions. It's not at all a "raw disk image", which I take to mean as the Posix disk device (e.g. /dev/disk0 or /dev/rdisk0).

Since CCC uses rsync, that also means that updates take only as long as necessary to copy any differences across. Files that didn't change aren't recopied. That's the same strategy that Time Machine applies.

You're correct that Time Machine will keep multiple versions of files, which CCC won't do by default. CCC can be told to preserve prior versions of files, but it won't keep as many past versions as TM, and not by default.

You can enable whole-disk encryption on a CCC target disk, so both can make encrypted backups. CCC can also write to a disk image, and disk-images can be encrypted.

 

[SteveJobzniak]

@Weaselboy: Ah okay your data needs are very modest, then it makes sense.

I found a way to make the official Amazon S3 calculator work now (http://calculator.s3.amazonaws.com/index.html). Disable adblocker, that was the problem for me last time so I used some unofficial calculator.

Alright here's what $4.99 (Amazon Cloud Drive's Unlimited cost) would really buy you via S3:

- Standard Storage: 170 GB.
- or Reduced Redundancy Storage: 205 GB.
- or Glacier (with 2 GB of standard storage for metadata): 390 GB. But the retrieval times are 4 hours per file, and costs a lot of money to retrieve files. So Glacier is terrible!

For very modest needs (less than 170 GB of super-secure storage, or 205 GB of less redundant backups), S3's reduced or standard makes the most sense!

For me, I've got about 100 GB of home-folder data, and 400 GB of music production reinstaller files to back up. So I am currently storing 500 GB on Amazon Cloud Drive Unlimited for $4.99/month, which is equivalent to S3's Reduced Redundancy Storage (it is the exact same back-end, and I get connected to S3 servers to upload the data). With S3 that would have cost me $12.04/month. I'm never touching Glacier with its 4 hour retrievals, but even Glacier would have been more expensive.

Well this settles it. Anyone with small needs (and who don't expect their needs to grow) should choose S3. Thanks for the correction!

--------

@chown33: Oh! I always assumed "bootable backups" and "clone" meant Carbon Copy Cloner took a snapshot of the exact sectors on the drive, since it would need the boot sector.

I use rsync manually when I sync things between different drives. It's a good command, but I wouldn't use it for backups.

Carbon Copy Cloner sounds a bit less terrible now, but still worse than Time Machine. What if a file was corrupted and you didn't notice until months later? That's what I love about Time Machine and Arq.

 

[chown33]

... What if a file was corrupted and you didn't notice until months later? That's what I love about Time Machine and Arq.

That's what CCC's "checksums" are for:
https://bombich.com/kb/ccc4/advanced-settings
Heading: Find and replace corrupted files

CCC normally uses file size and modification date to determine whether a file should be copied. With this option, CCC will calculate an MD5 checksum of every file on the source and every corresponding file on the destination. CCC then uses these MD5 checksums to determine if a file should be copied. This option will increase your backup time, but it will expose any corrupted files within your backup set on the source and destination. This is a reliable method of verifying that the files that have been copied to your destination volume actually match the contents of the files on the source volume.
...​

AFAIK, Time Machine alone will not do this, nor anything like it. If a backed-up file's contents become unreadable (which CCC can detect by hashing), and the file itself isn't changed (same mod-date etc.), then TM will keep producing hard-links to the unreadable file in its backups (because mod-date and hard-links is how TM works). Eventually, TM will "age out" older copies of the file (presumably readable), and only the unreadable file and its hard-links will remain, except for TM's longer-term backups. At that point, TM's most recent backups will be corrupted, with no warning or error message that this has happened.

If an original file becomes unreadable, and TM doesn't see a change, then the original is corrupted, but TM's backup will still be readable. The same applies to CCC, so both are equally capable of providing an uncorrupted backup, for whenever the most recent backup took place. If one is only doing monthly CCC backups, then only the most recent monthly will be available.

 

[Gixene]

Arq is totally incredible. And yes it backs up your home folder by default but you can add other folders and drives too. Easily.

- Extremely well architected and efficient. Uses the "commit history" concept from Git to do extremely safe and stable backups, with full revision histories.
- Open file format and open-source tools exist for decrypting backups, so you are never stranded with proprietary backups that can't be recovered if a company goes out of business.
- Supports all major cloud storage providers. So you can choose exactly the storage plan you need.
- Full encryption so that only you can recover the files. Choose a strong password and you're safe!
- You can back up as many computers, users and internal or external hard drives as you want.
- No monthly fees except what your cloud provider charges. This is in contrast to "BackBlaze" and other such services which average $5/month per backed up user for the rest of your life.
- You can search for files/folders and restore them (via the Arq GUI), and can see all historical versions it has backed up of that file/folder.
- Extremely fast and efficient backup daemon, unlike "BackBlaze" and other such garbage services which have bloated, memory-intensive software which slows down your computer.
- Only uploads changes in files, instead of sending the whole file all over again if its contents change. This means it is very bandwidth efficient.
- You can back up external hard disks by connecting them, adding them to Arq, and ticking the 'Skip if volume not mounted' box. The data will stay online forever. This is in contrast to "BackBlaze" and other such services which delete disconnected external drive backups after a few months.
- It does a monthly full verification to check that all backed up data still exists online, and it uploads anything that's missing (which it never is, but it's good to have the feature).
- Arq calculates hashes (aka checksums) for all file chunks it uploads, and only stores identical data once (multiple files with the same content will all point to the same "data" chunk online).
- On Amazon Cloud Drive, it (thanks to my feature request actually) verifies that each chunk was 100% correctly uploaded by checking Amazon's response (which has an MD5 hash) to know that the chunk was properly uploaded and not corrupted in transit. If it was corrupted, it re-uploads that chunk to Amazon.

Personally I use Arq with Amazon Cloud Drive Unlimited, and it truly is "unlimited". Other Arq users have backed up amounts such as 5 terabytes and 10 terabytes without getting complaints from Amazon. And Amazon Cloud Drive uses Amazon S3 as its storage back-end, so it's pretty much as good as S3, but without any storage costs. It has lower backup redundancy than S3 but so what? Arq verifies the data monthly, and Amazon keeps their storage drives in top shape and replaces any hard drive that's about to fail long before it fails. (@Weaselboy: You should really consider switching from S3 to ACD, unless you're using so little data that your S3's storage+transit costs are cheaper than ACD's $4.99/month. At something like 60 GB you've exceeded ACD's monthly cost if you're using S3. I use about 500 gigabytes which would cost about $35 per month via S3, and I will be needing even more when I back up even more computers later, which is why Amazon Cloud Drive Unlimited is perfect for me).

I bought Arq at Version 5, which was a major rewrite where he changed from a per-computer license to a single global license that you can use on all your computers. It's only 1 guy coding it, Stefan Reitshamer, but he's been doing this for years and is extremely talented, thorough and devoted. I foresee myself using this for as long as he keeps making it, and I bet that he'd sell the product to another company so that someone else can continue development if he ever reaches a day where he's too old or sick to carry on. But as of right now, he's pretty young and this is his only source of income and there's no sign of him ever stopping.

Therefore, I bought Arq 5 ($49.99) with the "Lifetime" license (which costs $29.99 extra) for a total of $79.98. That'll keep all of my computers safe for years and is totally worth it. It covers all versions he will ever release. Otherwise you'd pay $24.99 again per each major version upgrade. So the lifetime license has paid for itself by the time Arq 6 is out. And then it just keeps on giving.

Arq is the BEST, fastest, and most powerful online backup solution. Your data, encryption keys and cloud storage solutions all belong to YOU instead of some faceless corporation. I recommend that you buy the lifetime solution and get used to Arq. It's going to serve you well.

As for Time Machine: I use that too, for local network backups with instant restores. To me, Arq is only for the "okay my house just burned down or was burglarized and I have lost all data and decades of work and personal memories" situation. And for that, it gives me TOTAL peace of mind. I'll never lose a project or backup.

As for Carbon Cloner: I never used it. I don't see the point of creating a totally raw image of a disk, which is an image that take ages to create and quickly grows out of date. That's masochistic. I'd rather pick up the data on that disk via Time Machine and Arq. And if something gets screwed up, I do a full OS restore/reinstall via Time Machine, or I reinstall the entire OS on a new machine and restore particular files via Time Machine or Arq. Both of those solutions let me choose the exact point in time I want to restore from.

Time Machine + Arq is a great combo. But I could even survive with only Arq, because I don't really need to recover files with Time Machine very often, and I *could* therefore live with the extra time it would take to download the backed up files from the cloud.

I've actually been thinking of turning off Time Machine and only using Arq, but I have an AirPort Time Capsule 2TB so there's no point turning it off. And it's nice to have the option to do a full OS reinstall via Time Machine.

In short: Ditch Carbon Copy Cloner. Start using Time Machine (backing up to the disk you used to use for your carbon clones) and Arq instead. Because Time Machine does everything Carbon Copy did but much faster, more efficiently and with more granularity and automatic updating of the backup so that it's never outdated. And you can also encrypt your Time Machine backup with a password (so that it stores everything using AES encryption), which is much safer than Carbon Copy Cloner!

I truly admired your reply until you mentioned Time Machine when I clearly stated to "Please, respect this decision and leave it at that." Time Machine isn't for everyone. Neither is Safari, KeyChain, iCloud, Apple Music, Spotlight, etc. etc. It's important for Mac users to identify whether Apple's built in applications meet their computing needs. Rarely does Apple's built in applications not meet the needs I require, but Time Machine is one.

6 years ago I purchased a personal computer for the first time, a MacBook. These past 6 years, I did not backup data. Mainly because I had never heard of this concept until 2 months ago. Thankfully, I did not lose any data to which I am grateful. I searched for a solution and consulted the MacRumors community. Time Machine and Arq were among the recommendations I received. However, I wanted to focus finding one backup solution at the time.

Time Machine may have very well been an excellent backup solution in 2006 when it was first released. Though Apple has clearly neglected Time Machine over the 10 years. On top of that, Apple has indeed abandoned AirPort Express, Extreme and Time Capsule based on a press release earlier this month by Apple. Time Machine along with Apple's AirPort accessories have a lot of potential. Apple could replace SSDs with the outdated HDD in AirPort accessories. Apple could have created an option to backup Time Machine to iCloud. Except Apple has left Time Machine in the 2000's, pun intended. Apple may very well be pushing Mac users to use iCloud, look at iCloud optimization feature and iCloud Desktop and Documents sync added in Sierra.

Though why should I adopt a technology Apple is neglecting? 10 years later, Apple users have more options as technology has progressed. Some of these options serve a more beneficial purpose than what Time Machine can provide. For me, Time Machine services no purpose at all. I have never deleted data that I did not intend to. If I physically move data once to Move to Trash and then Empty Trash, then I did not need that data. Now you can restore the deleted data via iCloud. So Apple requires 3 simple requirements to delete data. I have never desired to return to a prior OS version. I adopt Apple's OS updates with confidence. I provide Apple with diagnostic & usage data to help Apple improve its products and services. I believe this is important as an Apple user to stand behind Apple to help Apple succeed.

Why would I place confidence in a backup solution that cannot be easily tested or verified? Time Machine does not allow users to verify the Time Machine's backup. This is what I believe is the reason Mac users have issues when Time Machine is needed to backup data and the data cannot be restored due to unforeseen issues. The integrity of the backup solution is the most important for the needs I require. Apple users should never rely on one backup solution. If Apple users could only have one backup solution, would it be Time Machine? For me, the answer is clear.

Carbon Copy Cloner provides the confidence that Time Machine cannot provide. Carbon Copy Cloner has a Checksum feature which verifies each clone I preform, to prevent cloning of corrupted data and will find the corrupted data to pinpoint the corruption. This feature is reassuring, but it does not compare to booting from the cloned drive once a month to ensure a successful backup. This is a peace of mind that Time Machine can not provide.

I have a 2016 MacBook Pro with Touch Bar & Touch ID. Spectacular machine. Before I had a 2010 MacBook White Polycarbonate. One of Apple's most iconic machines in my humble opinion. If the SSD in this MacBook Pro fails, I cannot replace the soldered SSD compared to the replaceable HDD secured with screws in the MacBook. If the SSD in this MacBook Pro fails, will I be in a financial state to afford Apple's service to replace SSD in this machine or even a replacement machine? If I'm unable to, I have a bootable clone on an external SSD that can allow continued use until I can afford to replace the SSD or this machine. Regardless, Carbon Copy Cloner's clone can use Apple's Migration Assistant to restore data like Time Machine for each situation.

I backup the clone once a month and move the external drive 7 miles from the location of this MacBook Pro. Time Machine would store a month of backups locally here on this MacBook Pro until I mount the external drive. This is not only risky, but a waste of SSD storage. Even more, Time Machine's backup is said to decrease the performance while it preforms. Once a day, I would have to endure this preformance impact and suffer the consequences once a month while it backs up to the external drive. Why would I retrieve and replace an external drive 7 miles from the location of the MacBook Pro once a month with Time Machine when I could use Carbon Copy Cloner's bootable cloning?

In short: Ditch Time Machine. Start using Carbon Copy Cloner (backing up to the disk you used to use for your Time Machine backups) and Arq for the versioning Time Machine provides. Because Carbon Copy Cloner does everything Time Machine does but faster (30 seconds using Thunderbolt 3), more efficiently such as a bootable backup and with checksum that can actually verify that your backup has integrity. And you can also encrypt your Carbon Copy Cloner backup with File Vault 2 (so that it stores everything using FileVault 2 encryption), which is much safer than AES encryption!

And yes, you are reading correctly. You can enable File Vault 2 to encrypt the cloned backups.

Thank you, Gixene
[doublepost=1480293579][/doublepost]

Fellow Arq user here and I'll try and answer your questions.

1. Like you mentioned, Arq by default only backs up the data in your home (user) folder. This is sufficient to save all your personal information and files. Arq does not backup things like applications or the OS itself.

2. Run the command below in Terminal to show the size of your users folder in GB. That should come close to the 14GB Arq has backed up. The Arq backup will be slightly smaller since it excludes some cache files.

du -hs ~/

3. The OS is not part of the Arq backup. You would first need to reinstall the OS, then you could restore your personal data using Arq. The Arq documentation has a section on how to do that.

https://www.arqbackup.com/documentation/

4. What you are seeing is normal. Arq breaks up your data into encrypted blocks and sends it to the backup server. It cannot be viewed like you can view files on something like Dropbox for example.

5. Like you discovered, Arq does versioning. It also does a monthly file verification similar to checksum. If you really only have around 14GB of user data, Amazon S3 would be much cheaper for you. Heck even if you allowed S3 to store 2X the data you have for versioning, it would only cost you less than one dollar a month vs. five dollars a month for Amazon Cloud Drive. I have about 30GB of user data on S3 with Arq and I set the Arq purge budget at $1.40. So that gives me about 46GB on S3 (140/3 cents a GB=46). That give me plenty of versioning going back a few months. I have Arq set to run once a day. To me hourly is more that I need and IMO just racks up storage space for unnecessary versioning.

Hope this helps. Let me know if you have more questions.

Thank you Weaselboy for taking the time to reply.

I ran the Terminal and below is a screenshot. To be clear the original size of the backup was 14.003 GB. There have been 10 backups since then ranging from 2.711 GB to 3.951 GB. Though Amazon Cloud Drive shows 1.6 GB used. So if I understand you correctly, it did not backup the 14.003 GB, only the 2.7 GB shown in Terminal. If this is not correct, please advise.



I'm glad you mentioned that Arq does monthly file verification. This was unbeknown to me, but very reassuring and places more pride and trust in using Arq. I see in Arq's Preferences that it has Validate backup data every 60 days. Is this the verification you are referring to? If so, it seems Arq's default is set to 60 days and I will change it if this is what you were referring to. It seems you still recommend versioning despite Arq's checksum feature? I'm unsure what Arq did during a backup a couple of hours ago, however it did show an error for Safari Preferences. Was this Arq's checksum feature? I was able to search for this issue here on MacRumors and actually found your solution using Terminal, Finder and deleting the file. Then rebooting and opening Safari. This was the first time I had ever used Terminal. Thankfully, I was able to backup successfully after this. Very reassuring the Arq is looking at something as little as Safari Preferences, showing nothing slips through Arq's backup! You are of great help here on MacRumors!

I would also agree that Arq's hourly backups are unnecessary. I simply don't have large fluctuations of data. I will change this to daily, even then this is quite unnecessary. However, I will use this option for the time being.

 

[Weaselboy]

I ran the Terminal and below is a screenshot. To be clear the original size of the backup was 14.003 GB. There have been 10 backups since then ranging from 2.711 GB to 3.951 GB. Though Amazon Cloud Drive shows 1.6 GB used. So if I understand you correctly, it did not backup the 14.003 GB, only the 2.7 GB shown in Terminal. If this is not correct, please advise.

If you used the default backup settings, it would only ever backup that 2.7GB users folder. It would backup the whole thing the first time, then only additions and changes afterwards.

I see in Arq's Preferences that it has Validate backup data every 60 days. Is this the verification you are referring to? If so, it seems Arq's default is set to 60 days and I will change it if this is what you were referring to.

Yes, that is the validation I am talking about. It has been a while since I set this up, so I may have changed it to 30 days and forgotten.

I have Arq setup to email me after each backup. That email shows how much was backed up and any errors. There is also an option to show a list of files backed up in that email if you want to see what is going on. Here is a snap of what that email looks like.

Backup session for /Users/xxx started on November 27, 2016 at 9:00:11 AM PST
Found completed backup record for /Users/xxx
Saved backup record for /Users/xxx
Scanned 24.488 GB (70490 files)
Uploaded 157.4 MB
Backup session for /Users/xxx ended on November 27, 2016 at 9:01:22 AM PST

 

[SteveJobzniak]

In short: Ditch Time Machine. Start using Carbon Copy Cloner (backing up to the disk you used to use for your Time Machine backups) and Arq for the versioning Time Machine provides. Because Carbon Copy Cloner does everything Time Machine does but faster (30 seconds using Thunderbolt 3), more efficiently such as a bootable backup and with checksum that can actually verify that your backup has integrity. And you can also encrypt your Carbon Copy Cloner backup with File Vault 2 (so that it stores everything using FileVault 2 encryption), which is much safer than AES encryption!

And yes, you are reading correctly. You can enable File Vault 2 to encrypt the cloned backups.

Thank you, Gixene

Thanks for the recommendation. I had a lot of years-old misconceptions about Carbon Copy Cloner being a super slow "1:1" raw disk clone tool. If I ever decide to change my on-site (at home) backups to something other than Time Machine, I will be trying CCC.

Anyway, for off-site (cloud) backups there is nothing better than Arq. I spent days researching it and other online backup solutions. And nothing else is as fast, safe, flexible and efficient as Arq. It's blazingly fast and doesn't impact my computer's performance.

I would 100% suggest the lifetime license option, by the way. That's what I picked because Stefan is a very active developer and the lifetime choice has paid for itself at Arq 6's release and then it just keeps on giving. He usually releases a major version every 1-2 years, and backups are something you want for life.

 

[Weaselboy]

https://twitter.com/arqbackup/status/803224056900292609

Looks like Arq is still on sale at 40% off for Cyber Monday if anybody is interested.

 

[SteveJobzniak]

https://twitter.com/arqbackup/status/803224056900292609

Looks like Arq is still on sale at 40% off for Cyber Monday if anybody is interested.

Oh. In my post above I actually shared my personal 40% link from the promotion email I got today, but I edited and deleted it from my post because it said "Coupon expired" when I checked. But now I see that even that twitter coupon is expired so I guess it wasn't just mine.

I am sure he will resolve it. This is a great chance for someone to buy Arq5+Lifetime. I suggest people keep an eye on when the Twitter coupon is fixed.

 

[campyguy]

I am sure he will resolve it.

It was fixed a couple of minutes ago.

 

[SteveJobzniak]

It was fixed a couple of minutes ago.

Thanks for letting people know! I hope @Gixene gets in on this deal. 40% discount is a great offer!

 

[Gixene]

Thank you Weaselboy, SteveJobziak and campyguy for informing me of the discount! I have never bought a lifetime upgrade before for any Mac application, as I worry about the future of the application and the developers intentions. However I love the application and seeing that it's the 5th version since 2009, I decided it would be a smart decision. Plus SteveJobzniak's words about the developer is encouraging.

If you used the default backup settings, it would only ever backup that 2.7GB users folder. It would backup the whole thing the first time, then only additions and changes afterwards.



Yes, that is the validation I am talking about. It has been a while since I set this up, so I may have changed it to 30 days and forgotten.

I have Arq setup to email me after each backup. That email shows how much was backed up and any errors. There is also an option to show a list of files backed up in that email if you want to see what is going on. Here is a snap of what that email looks like.

Understood. I changed the validation to 30 days. Is there any benefit to the emails? I noticed that Arq informed me of the Safari Preferences error in the application itself. So long as Arq does this, I see no reasoning to keep the emails. Unless I could set up a rule to move these to an archived mailbox or file folder on the MacBook Pro itself. Do you have a rule set up for this to keep these emails organized? How do you do this?

From what I understand this far, I would agree that S3 would be more cost effective than Amazon Cloud Drive. The past two days, using Arq has increased Amazon Cloud Drive storage from 1.4 GB to 1.7 GB. Based on this, it appears that I would have 4.5 GB of data after 30 days of Arq. Storing 3 months with versioning would have 13.5 GB of data. The cost of Amazon S3 would be $0.405 a month if this math is correct. Of course I understand that this is only 2 days of use with Arq, but perhaps it provides a general idea of the costs of using Arq to backup to Amazon S3.

However, I don't quite understand how the budget feature works. Example, if $0.405 provides 3 months of storage. Then is this the amount I would enter in the budget field or is there a different interface for Amazon S3 compared to Amazon Cloud Drive?

 

[SteveJobzniak]

@Gixene: I am happy we could help you! It's a great feeling to know that all of your personal files are safe even if your house burns down or there's a break-in or your main drive dies and your backup drive dies too while trying to restore, or any other situation that may arise when your house contains the only copies of your data.

With Arq, it's great to know that you can just install Arq on any new computer, log in to your cloud storage, and recover all of your invaluable files.

For your questions:

- I do not use emails. They're just a nice extra for the few people who want the information emailed to them for some reason. But if there are any errors, Arq pops up a notification on your screen anyway (very rarely I'll see an error because I deleted a file while Arq was backing it up, in which case it just says it couldn't read that file and skips it). Furthermore, the main GUI itself has a "!" sign next to any backups that have had errors. And you can always click the Arq menu bar icon and "View backup session logs" for a full history of all backups. So no, you don't need to manually keep track of archiving email logs. All logs are always stored in the app and I can in fact still see every hourly backup log since I installed Arq back in July.

- For the S3 storage question: Yes with your modest needs, you're best off choosing S3. See the price math here to see when Amazon Cloud Drive overtakes it in value: https://forums.macrumors.com/threads/arq.2017935/#post-23987777

- The "Budget" option in Arq is exactly what it says in the GUI:

- If you add an S3 storage there's even more control over the budget, such as choosing to move inactive files to Glacier storage etc. I hate Glacier though, so I recommend you stick with Standard S3 or Reduced Redundancy S3. Here's why: It takes 4 hours per file to restore Glacier files and it costs a ton of money: https://medium.com/@karppinen/how-i...d-from-amazon-glacier-6cb77b288c3e#.xq08jpd9t

- S3 Reduced Redundancy vs Standard:

(Source: https://aws.amazon.com/s3/faqs/#How_durable_is_Amazon_S3)

For your low amount of backup storage, even Standard is going to be super cheap.

- For people who need to store more than 170 GB of data in the cloud (and who are also reading this thread), you should choose Amazon Cloud Drive instead since it's literally built on top of S3, and offers the greatest value after you cross 170 GB of data:

I have had confirmation from Amazon that Amazon Cloud Drive is based upon S3 technology:

"Each file is stored within Amazon Simple Storage Service (S3); the same highly scalable, reliable, fast, data storage infrastructure that Amazon uses to run its own global network of web sites."

(Source: https://forums.aws.amazon.com/message.jspa?messageID=611889)

 

[Weaselboy]

However, I don't quite understand how the budget feature works. Example, if $0.405 provides 3 months of storage. Then is this the amount I would enter in the budget field or is there a different interface for Amazon S3 compared to Amazon Cloud Drive?

I just like the emails because it is an easy way to see how much has backed up and it there are any errors. But you don't have to do this like you mentioned. I don't keep them, I just delete each after I read it.

You use this screen in Arq for the budget. The way it works is on the day the "enforce budget" process is run, it will remove old backups to get you down below the budgeted amount. So in my example (160/3=~53GB) it will remove everything over 53GB. Then beginning from that moment forward, the amount drifts back up over 53GB again. My daily backup is usually around 300MB. So by the next budget enforcement I may have run up to 62GB or so on S3 (300*30=9GB) and the enforcement would pull it back down to 53GB.

 

[SteveJobzniak]

Worth noting: If you schedule your daily backup to something like midnight, and your machine is off, it will do that backup immediately as soon as you start your computer again.

However, I do hourly backups because I prefer it incrementally uploading ~8-40 megabytes per hour instead of waiting all day to do it all as a massive gigabyte chunk later. And I like the granularity in hourly file revision histories whenever I need them.

I use the "Thin backups from hourly to daily to weekly" option to not be overwhelmed with hourly backup entries. It works as follows:

https://www.arqbackup.com/documentation/pages/thinning.html

Backup Storage Policy
Arq retains backups in a similar manner as Apple's Time Machine.

Arq automatically makes incremental backups every hour (if you've set the schedule to hourly backups). It "de-duplicates" your files when it stores them at the destination, so the same file contents are never stored more than once.

If you check "Thin backups from hourly to daily to weekly" in the Advanced preferences pane, Arq saves the hourly backups for the past 24 hours, daily backups for the past month, and weekly backups until the amount of data stored at the destination reaches the budget you've set in the Preferences.

NOTE: Arq won't "thin" your backups (drop hourly backups older than 24 hours etc) if the Arq main window is open. It waits because you may be in the midst of browsing a backup that's scheduled to be dropped.

 

[Gixene]

Does one believe Amazon S3 will continue to have a free trial 3 months from now? I am using Amazon Cloud Drive's 3 month free trial. I would prefer to move from Cloud Drive to S3, but can't decide if I should continue to take advantage of the current trial.

Also, is it recommended to set Arq the user's home wireless network? Is there a possible security issue when using a public wireless network while Arq is backing up data? I understand it encrypts this data, but I'm unsure if the data is still at risk in a pubic wireless network.

@Gixene: I am happy we could help you! It's a great feeling to know that all of your personal files are safe even if your house burns down or there's a break-in or your main drive dies and your backup drive dies too while trying to restore, or any other situation that may arise when your house contains the only copies of your data.

With Arq, it's great to know that you can just install Arq on any new computer, log in to your cloud storage, and recover all of your invaluable files.

For your questions:

- I do not use emails. They're just a nice extra for the few people who want the information emailed to them for some reason. But if there are any errors, Arq pops up a notification on your screen anyway (very rarely I'll see an error because I deleted a file while Arq was backing it up, in which case it just says it couldn't read that file and skips it). Furthermore, the main GUI itself has a "!" sign next to any backups that have had errors. And you can always click the Arq menu bar icon and "View backup session logs" for a full history of all backups. So no, you don't need to manually keep track of archiving email logs. All logs are always stored in the app and I can in fact still see every hourly backup log since I installed Arq back in July.

- For the S3 storage question: Yes with your modest needs, you're best off choosing S3. See the price math here to see when Amazon Cloud Drive overtakes it in value: https://forums.macrumors.com/threads/arq.2017935/#post-23987777

- The "Budget" option in Arq is exactly what it says in the GUI:

View attachment 675175

- If you add an S3 storage there's even more control over the budget, such as choosing to move inactive files to Glacier storage etc. I hate Glacier though, so I recommend you stick with Standard S3 or Reduced Redundancy S3. Here's why: It takes 4 hours per file to restore Glacier files and it costs a ton of money: https://medium.com/@karppinen/how-i...d-from-amazon-glacier-6cb77b288c3e#.xq08jpd9t

- S3 Reduced Redundancy vs Standard:

View attachment 675176

(Source: https://aws.amazon.com/s3/faqs/#How_durable_is_Amazon_S3)

For your low amount of backup storage, even Standard is going to be super cheap.

- For people who need to store more than 170 GB of data in the cloud (and who are also reading this thread), you should choose Amazon Cloud Drive instead since it's literally built on top of S3, and offers the greatest value after you cross 170 GB of data:



(Source: https://forums.aws.amazon.com/message.jspa?messageID=611889)

Arq seems proactive in alerting users in regards to errors. Like you mentioned, Arq provides backup session logs in the application. I will continue to opt out of the email feature. The calculator you provided is certainly detailed. Perhaps a bit more complex based on the knowledge I have. However, I can clearly understand that Amazon Cloud Drive would be more costly compared to Amazon S3 for the storage needs I desire.

I've done a bit of research on the different services Amazon offers. I never would have expected Amazon to be (from a novice opinion) one of the most competitive cloud providers. I appreciate the options Amazon provides, but is quite complicated. I have files that I could never need for months, years even. Essentially, it's a backup solution and that Arq would be a 2nd option to backup, Carbon Copy Cloner being the 1st. Though I would be more than willing to pay $0.03 per GB to store with Amazon S3. If there were a larger margin per GB, between Amazon S3 and Glacier, then I may consider it. However, you can't put a price on the convince of 'on demand' storage. With this said, I will use Amazon S3 instead of Glacier.

I appreciate you explaining the differences between Amazon Standard Storage and Standard - Infrequent Access Storage. This greatly helped with understanding. As far as redundancy, I would opt for the Standard Storage. Like Glacier, I feel there isn't significant cost savings compared between these options. In fact, I would pay more for more redundancy. I realize that Amazon S3 Standard Storage has a 99.999999999% durability. Is this an option? If so, is it worth paying more for this option?

I just like the emails because it is an easy way to see how much has backed up and it there are any errors. But you don't have to do this like you mentioned. I don't keep them, I just delete each after I read it.

You use this screen in Arq for the budget. The way it works is on the day the "enforce budget" process is run, it will remove old backups to get you down below the budgeted amount. So in my example (160/3=~53GB) it will remove everything over 53GB. Then beginning from that moment forward, the amount drifts back up over 53GB again. My daily backup is usually around 300MB. So by the next budget enforcement I may have run up to 62GB or so on S3 (300*30=9GB) and the enforcement would pull it back down to 53GB.

View attachment 675236

So it seems I would need to store 3 months of data to see how much is stored. Then decide a budget from there. It would be great if Arq provided a feature on how long to store each version.

Worth noting: If you schedule your daily backup to something like midnight, and your machine is off, it will do that backup immediately as soon as you start your computer again.

However, I do hourly backups because I prefer it incrementally uploading ~8-40 megabytes per hour instead of waiting all day to do it all as a massive gigabyte chunk later. And I like the granularity in hourly file revision histories whenever I need them.

View attachment 675243
I use the "Thin backups from hourly to daily to weekly" option to not be overwhelmed with hourly backup entries. It works as follows:

https://www.arqbackup.com/documentation/pages/thinning.html

Good idea, I have changed it to midnight. I've heard a rumor that the best practice is to backup after powering on the machine than before powering off. Since Arq will backup when I power on the machine and I typically do not use the machine after midnight unless it's the weekend, it seems this would be best.

 

[11201ny]

Arq + ACD.

Anyone having issues where certain files/folders create a situation where Arq freezes your Mac, and causes it to restart (endlessly if the Arq Agent is running)?

In my initial 7.5TB upload, i have come across this issue 3 times so far - i'm about 4TB uploaded. I'm sure it will happen again. So far I have "Excluded" the folders the problematic files have come from. Eventually i'll need to isolate the specific files.

Their support was 90% useless.

 

[flyingspur]

Have used Arq for quite some time, excellent! My opinion is AWS S3 and Google Nearline are
great for what you seek. I use both extensively along with Google regional. Beware, Amazon
Cloud Drive may bottleneck your up/down speed. @Gixene I'm sure your well on your way

 

[Weaselboy]

Anyone having issues where certain files/folders create a situation where Arq freezes your Mac, and causes it to restart (endlessly if the Arq Agent is running)?

I've never had that happen. Did you use the built in support tool to send logs along with your trouble report. I've used that a couple times and had good response.

 

[flyingspur]

I've never had that happen. Did you use the built in support tool to send logs along with your trouble report. I've used that a couple times and had good response.

Well, I do not know what happened to the rest of my reply above.

@11201ny
Arq support is stellar, get him on twitter and via email no problem with either if you have a problem.
Seems you have a more info thread here, https://arstechnica.com/civis/viewtopic.php?p=32809523#p32809523
I have no experience with BB. Not even sure if Arq is compatible with BB either. There might be your problem.

 

[11201ny]

Well, I do not know what happened to the rest of my reply above.

@11201ny
Arq support is stellar, get him on twitter and via email no problem with either if you have a problem.
I have no experience with BB. Not even sure if Arq is compatible with BB either. There might be your problem.

No, they are competing services. So far my experience with Arq has been okay, much better than BB. Stefan @ Arq had no experience comparable to mine to help me with. In the end, i just sort of figured it out myself. I was suprised at the lack of problem solving offered. So while I will say that Arq was definitely relatively quick to respond (within 12-24hrs), i was unimpressed at the lack of interest in solving the issue.

Regardless, i would still recommend Arq. Having a larger archive, I have certain needs that someone just looking to backup a boot drive, or a music folder doesn't have. For me, it is essential to know that I can rely on the backup (if push comes to shove). This is where I trust Arq the most - over the horror stories i have read about Backblaze & Crashplan.

"...I tried Mozy, Carbonite and Backblaze. They all offered "unlimited" size backup for a fixed fee, but the "unlimited" service had limits. I couldn't back up a network drive, for example. I couldn't keep backups of external drives reliably -- Backblaze promised to delete backups of external drives that haven't been connected for 30 days. They were vague about durability and availability of my backups. And then there was the risk of the backup storage provider closing up shop and me losing all my backups. I'd seen that happen before (Upline, Xdrive, Omnidrive, Digital Railroad)." - Stefan Reitshamer​

 

[SteveJobzniak]

In the end, i just sort of figured it out myself. I was suprised at the lack of problem solving offered. So while I will say that Arq was definitely relatively quick to respond (within 12-24hrs), i was unimpressed at the lack of interest in solving the issue.

Well, yeah. Stefan Reitshammer is an autistic ******* with zero people skills. He's kinda known for it. Lots of programmers are like that. I think he's recently hired some female to help him out with nicer customer support, though. He can be efficient if your problem is easy, but he personally treats customers as annoyances and hates hearing about bigger problems with his code. But he's still the best there is when it comes to backup solutions, hence why I am still a happy lifetime Arq customer and still gladly recommend it to others. Stefan isn't a very nice person, but his product is great. And that's what matters.

Arq: You can use any backup storage. You won't suddenly lose all backups just because something like BackBlaze ends up closing. And you have total control over your backups, unlimited computer backups, no restrictions placed on you by some company. And the Arq daemon is very efficient and was written with meticulous nerdiness for speed so that it doesn't bog down my computer (unlike most generic backup companies that just slapped together some slow, generic garbage in Java). It's so fast that it's usually faster for me to restore a file via Arq from Amazon Cloud Drive than to wait for my Apple Time Capsule (on the local network)'s hard disk to wake up and find the file (time capsules are dog-slow)! You can even keep permanent copies of deleted files online via Arq; just add a folder, sync it to the cloud, then click the "Detach" button in the GUI to mark that folder as "don't sync it from local anymore". Then delete the local copy. Now you have a permanent cloud copy that you can manually download at any time via Arq's GUI!

Most competitors won't even let you keep YOUR ACTIVE backups for 30 days if you disconnect an external drive, let alone keeping ANY file/folder FOREVER. That is part of why I love Arq! The detached storage feature is great for "cold storage" of unimportant files. You can upload 300 GB of movies or whatever that you want to archive. Then just detach that folder so that it stops syncing from your machine. And delete it locally. Voila. It stays safely encrypted and available online. And if you want something back, just download that again.