Arq

[Apple_Robert]

This works for me:
View attachment 1748960

Notes:
1. The @me.com address is really the same as then @icloud and the +arq is to make it easy to do Mail rules. The "To" address can be any valid email address anywhere (but you probably know that!)
2. The password is not the password for your Apple ID, rather it is an "App-Specific Password" which you create in your iCloud settings on this page https://appleid.apple.com/account/manage

Thank you for the help.

 

[Apple_Robert]

Arq just completed one of the hourly backups and 3 errors were noted. This is the first time I have had a backup with errors. It looks like it is related to Apple News for some reason. Maybe I am wrong.

Anyone else run into this? I emailed Arq support about this a few minutes ago.
-------------------

Errors:	3
Scanned Bytes:	6.888 GB
Scanned Files:	138,981
Uploaded Bytes:	48.1 MB
Uploaded Files:	264

Log:

25-Mar-2021 15:00:05 EDT Backup activity started
25-Mar-2021 15:00:05 EDT Arq version 7.2.3
25-Mar-2021 15:00:05 EDT macOS 11.2.3
25-Mar-2021 15:00:05 EDT Storage location: https://polycloud.crowdapis.com (S3:/m1mba2020)
25-Mar-2021 15:00:07 EDT Creating APFS snapshot for Data (/System/Volumes/Data)
25-Mar-2021 15:00:07 EDT Created APFS snapshot for Data (/System/Volumes/Data)
25-Mar-2021 15:00:07 EDT /: Backup plan changed; rescanning all folders
25-Mar-2021 15:11:35 EDT Error: /Users/robert/Library/News/com.apple.news.public-com.apple.news.private-production/referralItems/com.apple.news.widget/today--systemLarge-31C1C4F3-DCA3-4547-AA11-F8E6B827F96E/55C236B7-D211-4311-83D6-1E0BBDBEEC1A: Failed to read file status: No such file or directory
25-Mar-2021 15:11:35 EDT Error: /Users/robert/Library/News/com.apple.news.public-com.apple.news.private-production/referralItems/com.apple.news.widget/today--systemLarge-31C1C4F3-DCA3-4547-AA11-F8E6B827F96E/2284E471-E130-4512-AAC9-D4D4DDCFF8CA: Failed to read file status: No such file or directory
25-Mar-2021 15:11:35 EDT Error: /Users/robert/Library/News/com.apple.news.public-com.apple.news.private-production/referralItems/com.apple.news.widget/today--systemLarge-31C1C4F3-DCA3-4547-AA11-F8E6B827F96E: Failed to read file status: No such file or directory
25-Mar-2021 15:13:01 EDT /: Created a new backup record.
25-Mar-2021 15:13:01 EDT / (68 exclusions): 6,888,907,221 bytes, 115,824 files backed up
25-Mar-2021 15:13:01 EDT Total scanned: 6,888,951,278 bytes (138,973 files)
25-Mar-2021 15:13:01 EDT Total uploaded: 48,146,376 bytes
25-Mar-2021 15:13:02 EDT Errors occurred; not enforcing budget or removing unreferenced data
25-Mar-2021 15:13:02 EDT Removing APFS snapshot for /System/Volumes/Data
25-Mar-2021 15:13:02 EDT Removed APFS snapshot for /System/Volumes/Data
25-Mar-2021 15:13:02 EDT Backup activity ended
Questions? Please email us at support@arqbackup.com.

 

[Weaselboy]

I've had a similar error message but it was a a file related to a Mail message. In my case, I think what happened was I deleted a message as Arq was in the middle of backing up, so the file was not where Arq expected it to be, so it threw an error message. Try running another backup and see if you get the error message again. I'm thinking you won't.

 

[Apple_Robert]

I've had a similar error message but it was a a file related to a Mail message. In my case, I think what happened was I deleted a message as Arq was in the middle of backing up, so the file was not where Arq expected it to be, so it threw an error message. Try running another backup and see if you get the error message again. I'm thinking you won't.

Thanks. I will see what happens at 4Pm. By the way, I noticed at the bottom of the log "not enforcing budget or removing unreferenced data." Should we removed unreferenced data? I looked in Arq but, didn't see a tick box with the option to remove unreferenced data.

 

[Weaselboy]

You can set a storage budget here if you want.

The unreferenced file thing is the second checkbox here that starts with Keep deleted files...

It sounds like you have that box checked, so files that no longer exist on your Mac will stay on the backup on the server. If you uncheck it like I have it will remove "unreferenced" files that are no longer on your Mac. Really just a personal preference how you want to use this IMO.

 

[Apple_Robert]

View attachment 1749263

You can set a storage budget here if you want.

View attachment 1749264


The unreferenced file thing is the second checkbox here that starts with Keep deleted files...

It sounds like you have that box checked, so files that no longer exist on your Mac will stay on the backup on the server. If you uncheck it like I have it will remove "unreferenced" files that are no longer on your Mac. Really just a personal preference how you want to use this IMO.

Thanks for the explanation, WB. I looked at my settings and the "keep..." box is not ticked. By the way, the latest backup did not have any errors.

 

[gilby101]

By the way, I noticed at the bottom of the log "not enforcing budget or removing unreferenced data." Should we removed unreferenced data? I looked in Arq but, didn't see a tick box with the option to remove unreferenced data.

The Help says that unreferenced data is removed after every backup. And it is for mine. Maybe that line appears to show that Arq is not tidying up after this backup because of errors - it should tidy up next time. If the "not enforcing.." continues, ask Arq Support. That line in the logs was subject of a Support request from me early in the pre-release Arq7, and it was subsequently corrected - maybe still not fully correct. Also, please read my reply to Weaselboy, below.

The unreferenced file thing is the second checkbox here that starts with Keep deleted files...

It sounds like you have that box checked, so files that no longer exist on your Mac will stay on the backup on the server. If you uncheck it like I have it will remove "unreferenced" files that are no longer on your Mac. Really just a personal preference how you want to use this IMO.

I think that is not correct. Unreferenced data is objects/blobs on the backup server which hold data from files that no longer exist in any backup record. Files could have bee removed by thinning or by explicit deletion of a backup record/snapshot. As a consequence there may be empty objects on the server - these need to be deleted separately. The Help says:

If you select your backup plan in the main window and then pick “Remove Unreferenced Data…” from the menu, Arq uses its database to find all objects in the backup set that are no longer needed by any backup record, and deletes them.

Arq removes unreferenced data at the end of every backup activity, but if you delete some backup records and just want Arq to remove the data that become unreferenced as a result without doing a backup, pick “Remove Unreferenced Data…” from the menu.

The purpose of the "Keep deleted files in subsequent backup records" also in the help (but a bit obscure):

Instruct Arq to keep deleted files in subsequent backup records, so that a file deleted in the past is easily found in the latest backup record

I interpret that to mean that files deleted on your Mac (but still in an old backup record) will be listed in the current backup record - to make it easy to find old files. I have not used it.

 

[Apple_Robert]

The Help says that unreferenced data is removed after every backup. And it is for mine. Maybe that line appears to show that Arq is not tidying up after this backup because of errors - it should tidy up next time. If the "not enforcing.." continues, ask Arq Support. That line in the logs was subject of a Support request from me early in the pre-release Arq7, and it was subsequently corrected - maybe still not fully correct. Also, please read my reply to Weaselboy, below.

I think that is not correct. Unreferenced data is objects/blobs on the backup server which hold data from files that no longer exist in any backup record. Files could have bee removed by thinning or by explicit deletion of a backup record/snapshot. As a consequence there may be empty objects on the server - these need to be deleted separately. The Help says:

The purpose of the "Keep deleted files in subsequent backup records" also in the help (but a bit obscure):

I interpret that to mean that files deleted on your Mac (but still in an old backup record) will be listed in the current backup record - to make it easy to find old files. I have not used it.

Thanks for the informative reply.

 

[Apple_Robert]

By the way @gilby101 my latest backup (just now) did remove unreferenced data. It is curious why it didn't earlier, when I don't believe I had deleted anything.

 

[Weaselboy]

I think that is not correct.

Thanks for the correction.

 

[BigMcGuire]

I had this error once "25-Mar-2021 15:13:02 EDT Errors occurred; not enforcing budget or removing unreferenced data" and it eventually just went away. Not sure what I did to fix it. <shrug> but I remember it erroring out one day, seeing that and going Hmmm ok. Then poof didn't get it again.

One thing I've noticed is that if you set up a new Arq backup location - the Thumbnails agent in Mac OS goes high CPU for quite some time depending on how many files you've got.

Currently using OneDrive and B2. While the upload is about the same, the overall backup process seems a lot faster on B2 for whatever reason. I don't think OneDrive was designed to be a trillion small files backup service. But upload speeds are really good.

 

[gilby101]

Currently using OneDrive and B2. While the upload is about the same, the overall backup process seems a lot faster on B2 for whatever reason. I don't think OneDrive was designed to be a trillion small files backup service. But upload speeds are really good.

On speed: For me, with a 20 mbps uplink, OneDrive can be faster upload than B2 because of OneDrive's AU servers. But is more variable in speed. Overall much the same for daily backups. But if I do a 'validate', OneDrive is about 3 to 4 times slower than B2.
Arq seems best suited to destinations using S3-compatible APIs (e.g. AWS, B2 or Wasabi). Nevertheless it works pretty well with 'consumer' storage like OneDrive, GoogleDrive, etc.

 

[Apple_Robert]

How do you set up two different backups that run automatically like TimeMachine can do?

 

[Weaselboy]

How do you set up two different backups that run automatically like TimeMachine can do?

I have not done it, but it looks like you can just setup a new backup plan with whatever settings and destination you want and they will both run when you set them to.

 

[BigMcGuire]

I have not done it, but it looks like you can just setup a new backup plan with whatever settings and destination you want and they will both run when you set them to.

View attachment 1750817

Yep, that'll do it. That's how I did it:

Just remember to set the schedules on both - I do mine daily at 7am and 8am

 

[hallux]

I think I'm going to abandon ARQ. I have a Synology NAS, I set it up as a TM destination, I can then set up the NAS to back that up to Wasabi and I'll have my tiered backup system...

 

[gilby101]

I think I'm going to abandon ARQ. I have a Synology NAS, I set it up as a TM destination, I can then set up the NAS to back that up to Wasabi and I'll have my tiered backup system...

How sure are you about that?
On the NAS you have a TM sparsebundle. How well will that backup to Wasabi given that many bands (files in the sparsebundle) will change in each backup?
Recovery: You will need to recover the whole sparsebundle from Wasabi just to be able to restore a single file let alone recover a whole system! Do you know how to get TM to connect to the recovered sparsebundle?
Testing your scheme will be interesting.

 

[Apple_Robert]

Yep, that'll do it. That's how I did it:

View attachment 1750819

Just remember to set the schedules on both - I do mine daily at 7am and 8am

Thus far, I have noticed OneDrive is a lot slower than Polycloud. At first I thought it was due to the "backup and validate" option but, even with it off, OneDrive is still slower than Polycloud. It is taking a long time to complete my first backup to OneDrive.

 

[planteater]

I think I'm going to abandon ARQ. I have a Synology NAS, I set it up as a TM destination, I can then set up the NAS to back that up to Wasabi and I'll have my tiered backup system...

If the TM backup becomes corrupt for any reason, a second copy kept at Wasabi may inherit the corruption. With multiple backup media, onsite and off, as well as multiple methods feeding them, chances of recovery greatly improve. Just throwing it out as something to consider.

 

[hallux]

@gilby101 @planteater I hadn't considered that. Thanks

I was looking more to reduce resource usage on the Mac, but that's not a SUPER huge concern. I guess I'll keep it as is, with TM to the NAS and ARQ on the Mac going to Wasabi...

 

[Apple_Robert]

@BigMcGuire
I just signed up for the free 30 day trial of Wasabi (no credit card needed). It is very fast. I like that it offers Immutable backup, which Polycloud and OneDrive do not. Seeing how poorly OneDrive performed, I removed it from Arq. It was slower than a TM backup, which is not very practical, in my opinion. If Wasabi performs well over the next 28 days, I will stick with it and Polycloud for online backup, and use TM here at home.

Although the user documentation for Arq is lacking, I am very impressed with the program. Much better than TM.

 

[gilby101]

I just signed up for the free 30 day trial of Wasabi (no credit card needed). It is very fast. I like that it offers Immutable backup, which Polycloud and OneDrive do not. Seeing how poorly OneDrive performed, I removed it from Arq. It was slower than a TM backup, which is not very practical, in my opinion. If Wasabi performs well over the next 28 days, I will stick with it and Polycloud for online backup, and use TM here at home.

Re speed, everyone needs to do their own tests. So much depends on location of servers (and us). For me OneDrive is sometimes faster than B2 because servers are closer. And Wasabi (when I last used it) was significantly slower.

Glad Polycloud is working out well - I should try it!

Although the user documentation for Arq is lacking, I am very impressed with the program. Much better than TM.

With Time Machine to APFS, Apple now have a really good backup. Much more reliable that TM to HFS+. For local backup, I definitely would not consider Arq to DAS or NAS with TM support, only to a NAS (or other SMB server) without TM support.

 

[Weaselboy]

Purchase Arq

cloud.arqbackup.com

There is a sale 40% off Arq Premium if anybody is interested. Good until tomorrow.

 

[Apple_Robert]

Purchase Arq

cloud.arqbackup.com

There is a sale 40% off Arq Premium if anybody is interested. Good until tomorrow.

Thanks for the heads up. That is an excellent price. I went ahead and subscribed. Too bad we can't lock in that price. lol

 

[Apple_Robert]

For those interested, the Arq Premium (I just subscribed to thanks to Weasel) backup appears to be as fast (for me) as Wasabi.

I decided to ditch Polycloud even though speed was not a problem. Polycloud doesn't offer any console controls, which Wasabi does. It also doesn't offer 2FA, which Wasabi does, amongst other features. Another problem with Polycloud is the lack of any real customer support. I didn't like that. The price was right but all those things started bothering me after a few days.

I already made a reminder for next year, in case Arq offers another $36 special next year.