Arq

[BigMcGuire]

Thanks again! I sincerely appreciate your confirming my understanding is correct.

"Interesting setup": Using a 1TB SSD($$$), I also use Carbon Copy Cloner to backup my Mac daily. When the Mac is stolen or damaged, I can use the SSD to externally boot another Mac or clone the SSD to a new Mac. Downloading 500GB from ARQ backup data set from the cloud does not seem to be practical. So it is a real life scenario. I also do time machine backup on a hard drive because it is cheaper.

Definitely agreed - 500GB to cloud is expensive - much cheaper to do time machine or carbon copy. Very cool. I'm jealous of your setup . Definitely agree that CCC is a great way to back up a Mac for easy reproduction if lost or stolen. Best of wishes!

 

[dai-leung]

Definitely agreed - 500GB to cloud is expensive - much cheaper to do time machine or carbon copy. Very cool. I'm jealous of your setup . Definitely agree that CCC is a great way to back up a Mac for easy reproduction if lost or stolen. Best of wishes!

Thanks to u, users who use CCC and ARQ will know what to do when their Macs are stolen or damaged. I for one, greatly benefited from ur patient and kind explanation of how ARQ works. Last time, when I tried to use the SSD to externally boot another Mac, ARQ started to backup files. I thought ARQ's data set in the cloud was being corrupted, panicked and shut down everything. Since then, never have the courage to try the setup again. But now I have the confidence and know exactly what to do when disaster hits.

No, 1TB cloud storage is not expensive. I agree with u on having more than one cloud backup. I actually want to ask for info how to sign up B2 to do ARQ backup. The concern i have is that I will be then doing hourly ARQ backup to two destinations, will It slow down the Mac substantially? Is it possible to stagger the two backups such that one occurs on the hour while the one occurs 30 mins later?

The Mac I mentioned contains 26 thousands annotated scientific papers. If Mac is damaged or stolen, the owner will never recover professionally. Thus, the current price for cloud storage, license for ARQ and CCC are actually very affordable and reasonable and is an insignificant sum to saftguard the data. (1TB Samsung T5 SSD that I have has dropped to ~$340.)

Without the contributions and helps from ARQ users of this forum, I would not have known there is a program called ARQ, that is so easy to use even for a non computer person like me. Thus, I am immensely grateful to all these contributors and users who have helped me. With all ur helps and contributions , I am now confident that the contents of this Mac is safe.

 

[gilby101]

The concern i have is that I will be then doing hourly ARQ backup to two destinations, will It slow down the Mac substantially? Is it possible to stagger the two backups such that one occurs on the hour while the one occurs 30 mins later?

Yes. Each destination (I use two OneDrive accounts) has its own schedule. I have mine set to once a day at 10pm and 11pm, but you can do it hourly at a specified number of minutes after the hour.

 

[DarkIcarus]

I had a question involving Arq backups. What do people on here usually backup? As in the whole user folder, just documents & media, or only absolutely irreplaceable files like photos?

The reason I am asking is because with many cloud backup services you can, for example, usually delete files you no longer use/have. However, with Arq you cannot. I do fully understand that Arq will delete the oldest backup sets when you reach your preallocated quota. But I am just curious if users backup everything and/or files that changes often, or just the most important files that’s don’t change.

 

[BigMcGuire]

I had a question involving Arq backups. What do people on here usually backup? As in the whole user folder, Documents & media, or just absolute irreplaceable files like photos?

The reason I am asking is because with many cloud backup services you can, for example, usually delete files you no longer use/have. However, with Arq you cannot. I do fully understand that Arq will delete the oldest backup sets when you reach your preallocated quota however. But I am just curious if users backup everything and/or files that changes often, or just the most important files that’s don’t change.

I backup my OneDrive folder with Backblaze B2 set to delete NOTHING and no quota. (27GB). I have a folder with all of my Google Photos and videos (120GB) and an Apple Photo library of my iPhone's 14GB of photos that I've taken on my 8+ since getting it. At work, I backup my work folder and my work backup folder (50GB+). So I have almost 200GB saved via ARQ across 2-3 computers all to Backblaze B2.

I don't do any quota'ing because I want all versions/everything saved permanently. I do all my work in OneDrive (personal, college, work).

I never use Documents, etc.. folders. I do everything in OneDrive folder.

 

[Weaselboy]

I had a question involving Arq backups. What do people on here usually backup? As in the whole user folder, just documents & media, or only absolutely irreplaceable files like photos?

I do the entire ~/Users folder.

 

[DarkIcarus]

[doublepost=1524681964][/doublepost]

I do the entire ~/Users folder.

Do you use a quota?
[doublepost=1524682049][/doublepost]

I backup my OneDrive folder with Backblaze B2 set to delete NOTHING and no quota. (27GB). I have a folder with all of my Google Photos and videos (120GB) and an Apple Photo library of my iPhone's 14GB of photos that I've taken on my 8+ since getting it. At work, I backup my work folder and my work backup folder (50GB+). So I have almost 200GB saved via ARQ across 2-3 computers all to Backblaze B2.

I don't do any quota'ing because I want all versions/everything saved permanently. I do all my work in OneDrive (personal, college, work).

I never use Documents, etc.. folders. I do everything in OneDrive folder.

I realise you don't use Documents folder, etc. But does that mean your OneDrive folder has your documents, etc in?
I only ask this because phots, videos, music type file usually don't change that often, except to add new. But documents can get edited or changed often. Also some documents may only be temporary or not needed in the future. For example receipts, warranties, instruction manuals, etc. With my usage I see photos, videos and music to be things that I keep permeably. While some document type files would only be temporary. But because I cant delete old unwanted files from an Arq backup, I am wondering whether I should back them up to not.

 

[BigMcGuire]

[doublepost=1524681964][/doublepost]

I realise you don't use Documents folder, etc. But does that mean your OneDrive folder has your documents, etc in?
I only ask this because phots, videos, music type file usually don't change that often, except to add new. But documents can get edited or changed often. Also some documents may only be temporary or not needed in the future. For example receipts, warranties, instruction manuals, etc. With my usage I see photos, videos and music to be things that I keep permeably. While some document type files would only be temporary. But because I cant delete old unwanted files from an Arq backup, I am wondering whether I should back them up to not.

Yep yep, I have a folder inside of OneDrive where I keep all the docs I work on --- then in my Work folder I have a Work Logs folder, and in my personal folder I have a Daily Logs folder. Storage folder has a folder for bills, receipts, taxes, etc... Been doing this folder structure for 10+ years on various cloud platforms so it's very ingrained in how I do things.

Up to you - cloud storage is really cheap, so I was debating if I should upload my 120GB of photos/videos cuz I have them on disk and on Google Photos... I'm paying $.92/mo for 200GB of storage on Backblaze B2. If it was more than $10/mo I probably wouldn't have uploaded my photos.

 

[DarkIcarus]

[doublepost=1524683150][/doublepost]

Yep yep, I have a folder inside of OneDrive where I keep all the docs I work on --- then in my Work folder I have a Work Logs folder, and in my personal folder I have a Daily Logs folder. Storage folder has a folder for bills, receipts, taxes, etc... Been doing this folder structure for 10+ years on various cloud platforms so it's very ingrained in how I do things.

Up to you - cloud storage is really cheap, so I was debating if I should upload my 120GB of photos/videos cuz I have them on disk and on Google Photos... I'm paying $.92/mo for 200GB of storage on Backblaze B2. If it was more than $10/mo I probably wouldn't have uploaded my photos.

Ok, thanks. I also have a folder structure I have used for years and like yourself, its very much ingrained into how I do things. Personally I've always backed up music, photos, all documents, etc. I was just contemplating whether I should continue backing up all documents to the cloud if I cant delete them. As years go on the amount of documents I discard would be taking up cloud storage space and costing me, if ever so slightly. Where as before I could delete them in other cloud storage providers. However, in all other areas Arq seems a lot better. So please don't think I am trying to criticise Arq's inability to do this.

 

[Weaselboy]

Do you use a quota?

Yes... my users folder is about 30GB and I have Arq setup to keep 45GB on BackBlaze B2. That looks like it will give me archives back about six months.

 

[BigMcGuire]

[doublepost=1524683150][/doublepost]

Ok, thanks. I also have a folder structure I have used for years and like yourself, its very much ingrained into how I do things. Personally I've always backed up music, photos, all documents, etc. I was just contemplating whether I should continue backing up all documents to the cloud if I cant delete them. As years go on the amount of documents I discard would be taking up cloud storage space and costing me, if ever so slightly. Where as before I could delete them in other cloud storage providers. However, in all other areas Arq seems a lot better. So please don't think I am trying to criticise Arq's inability to do this.

Inside of Backblaze B2:

This isn't directly related to your comment - but yeah, I imagine I have a sizeable amount of document versioning saved in Backblaze. Although I only back up every 2 hours on my work machine and every hour on my home machine. Considering 200GB is only costing me $.92/mo (with me doing some downloading) - works for me, for now. Should only get cheaper too.

How interesting - I don't see any info on how it works with deleted files. My guess is it just keeps them up there. I spent some time looking and couldn't find anything. Since I prefer this, I'm happy with it, but I could see how someone who works with a TON of documents may not like this.

Definitely should be a feature request to Arq!

 

[DarkIcarus]

Yes... my users folder is about 30GB and I have Arq setup to keep 45GB on BackBlaze B2. That looks like it will give me archives back about six months.

Thanks for the reply. I think its useful keeping archives. I am also liking Arq due to versioning to help with any possible file corruption that may occur.

 

[BigMcGuire]

Yes... my users folder is about 30GB and I have Arq setup to keep 45GB on BackBlaze B2. That looks like it will give me archives back about six months.

When my new Mac comes, I'll most likely do the same thing as you - backing up the Users folder. Thanks for the info!

 

[DarkIcarus]

Inside of Backblaze B2:

This isn't directly related to your comment - but yeah, I imagine I have a sizeable amount of document versioning saved in Backblaze. Although I only back up every 2 hours on my work machine and every hour on my home machine. Considering 200GB is only costing me $.92/mo (with me doing some downloading) - works for me, for now. Should only get cheaper too.

How interesting - I don't see any info on how it works with deleted files. My guess is it just keeps them up there. I spent some time looking and couldn't find anything. Since I prefer this, I'm happy with it, but I could see how someone who works with a TON of documents may not like this.

Definitely should be a feature request to Arq!

As far as I'm aware, Arq does all versioning. This is why in one of the last few updates Arq changed that when you create a new B2 bucket Arq automatically sets it up with the option "Keep only the last version of the file". So if you don't have a quota set up tin Arq it will not delete any thing you backup at all. If you do set a quota in the Arq app it will delete the oldest backup record when it meets that quota. There is information in the documentation on the Arq website. I wasn't sure if I can post a link or not.

 

[BigMcGuire]

Just got my new MacBook Pro a few days ago. Followed @Weaselboy and put my whole user folder under Arq. Wow is that nice. I didn't realize that would include everything. Before, I'd have to locate DayOne's data files - back those up, and several other "apps" that I wanted data backed up from. Well just doing the whole user folder saves all that. REALLY nice.

 

[alexandero]

Dear all, what are your suggestions for backing up two laptops with a very slow internet upstream connection (7.5Mbps)? If the upstream is maxed out, that equals slightly above 3GB per hour. The two laptops each have a 50-100GB iTunes library, 20-30GB Pictures library, 10-20GB Mail library, and 5-10GB documents. Plus I have ~100GB of movies I'd like to backup to somewhere too.

Do you suggest a different (but also automated) backup strategy for iTunes, Pictures and movies, to create a significantly smaller Arq upload volume?

As I plan to also purchase a Synology or QNAP NAS, but I'd really like my Music and Photos saved offsite too (plus: I don't really trust Time Machine). But with a ~150GB initial Arq upload and a 3GB/hour speed I'm looking at net 50 hours upload. Considered laptops don't run 24/7 and are moved from home to work each day, that might result in weeks of initial upload per laptop – which is why I was hoping that I could use Arq to backup to the NAS and have a script on the NAS copy that data to S3/B2/…, but in a support email the Arq developer wrote that Arq uses a different file format for NAS backups than for S3, so that does not seem like a good idea.

Is keeping each laptop run uninterruptedly for 2+ days doing the initial upload the only option? Is it possible to start with backing up ~/Documents only, and then add folders (~/Desktop, ~/Library, ~/Music,…) stepwise every few days? Or is there a much smarter place to keep media files like music, photos and movies?

Thanks!

 

[jameshenderson]

Hi all,

I periodically get errors such as this one below backing my Mac using Arq to B2:

Error: ~/Library/Application Support/Plex Media Server/Media/localhost/0/d6863a3fac7f4caf8f81b066a97c7b9908f7976.bundle/Contents/Indexes/index-sd.bif: error reading request: java.net.SocketTimeoutException: Read timed out​

[snipped]

By the way, this was "fixed" in v5.11.3.3 - the error is on BackBlaze's end (they acknowledged the problem but haven't yet addressed it as far as I know). The update basically gets Arq to retry on seeing this error until it works.

 

[dai-leung]

I have been using a 1TB Dropbox account to backup 500GB Mac data. I set the ARQ budget to 800GB and found Dropbox was frozen because ARQ backup dataset exceeded 1TB. I had to delete the dataset to unfreeze Dropbox. It had happened three times. ARQ support could not find out what I had done wrong.

I just found out that the default value for the budget enforcement time is 30 days! In order for the budget to be useful, it should be set to 1 day. Or better yet 1 day to be done in early AM.

 

[AVonGauss]

Deleted.

 

[dai-leung]

The budget should really just be keeping it from holding on to prior version indefinitely rather than a hard limit with significant pressure.

Thanks for explaining the purpose of budget to me.

I had set the budget enforcement time to 1 day. Do u think I would run into unexpected problem?

I can see that if the budget is exceeded, ARQ will delete some old files from the backup dataset to lower the number of GB in the dataset. This will create a space between the dataset GB and the budget. Indeed, if the new space created is small, for example just 1MB below the 800GB budget, then any new file created, will exceed the budget again and another old file will have to be deleted. Constantly deleting old files to free up space is definitely undesirable.

Just wondering if u know when the budget is reached, how much space will be created by deleting old files?

It seems that no matter what storage amount one subscribes, he will eventually reach that limit. Do u know if there is a recommendation? For example for 1TB of data, one should have 3TB of storage.

 

[AVonGauss]

Deleted.

 

[dai-leung]

For ARQ, how much cloud storage space will be required for a given amount of data in the Mac and how the budget works are interesting questions.

Through reading articles in the Internet, this is my understanding of how ARQ work: (1) after compression and encryption, ARQ will reduced the Mac data to about 60% of its original size (amount of data stored in the cloud); (2) only new changes will be backuped. That is, if I change a 1TB file by 1MB, only 1MB(or 60% of 1MB) will be backuped to the cloud. Further, if there is no new change, the same file will not be backuped up again (deduplication).

Thus, it seem that it is the net data growth rate (data increase due to new files +new changes minus deleted data) per month that should decide the needed cloud storage space.

As an example, if my initial data in the Mac is 500GB and my net data is growing at the rate of 10GB per month (or 120GB per year). Then the initial ARQ upload will be only 300GB. After 12 month, my Mac data will increase to 620GB and the corresponding size of the ARQ dataset will be 372GB. So a budget of 800GB would seem to be more than sufficient. (So very puzzled why I exceeded 1TB storage space 3 times in 8 months.)

Eventually the budget limit is reached. Assume ARQ could delete old files corresponding to 10% of the budget (that is frees up 80GB of space in the cloud), this will give me another year before the backup dataset will exceed the budget again. Thus, 800GB of storage space will be more than sufficient for me.

I am not a computer person and I hope someone can explain how ARQ actually work.

 

[Weaselboy]

As an example, if my initial data in the Mac is 500GB and my net data is growing at the rate of 10GB per month (or 120GB per year). Then the initial ARQ upload will be only 300GB.

I have not noticed that Arq compresses data at all. I use Arq to backup my users folder of about 30GB, and the initial Arq backup was just under that due to Arq skipping some cache files I think. Then as you said subsequent backups include new files or changes.

With your example given, I would think 1TB of online space would be plenty, assuming you don't have a lot of changes and new files all the time.

 

[dai-leung]

I have not noticed that Arq compresses data at all. I use Arq to backup my users folder of about 30GB, and the initial Arq backup was just under that due to Arq skipping some cache files I think.

Thanks for clarifying the questions I posted.

When the data was small, the reported ARQ's compression ration was 16/40 or 40%.
"The degree of compression is nearly equal for both products, since in both cases the initial 40 MB of data was compressed to 16 MB."
Source: https://www.cloudberrylab.com/resources/comparisons/cloudberry-backup-vs-arq-backup.aspx
See also: https://www.arqbackup.com/blog/arq-5-massively-faster-backup-and-restore/

Not sure why when the data was 30GB, then the compression ratio became near zero.

 

[dai-leung]

How to block ATT hotspot?

I turned on ATT hotspot in an area without wifi to do work on my Mac without realizing that ARQ was doing backup in the background. Later found that ARQ had consumed a sizable amount of my data plan.

Is there a way I could select "backup from all available wifi networks" except ATT hotspot? That is to have a blocked wifi network list available.