Arq

[SteveJobzniak]

@dimme Chances that anyone on this forum has restored a cloud backup via hard drive shipping are less than 0%.

You'll have to ask Arq support. The question to ask is: If your cloud provider ships a hard disk with the entire contents of the Arq cloud folder, can that hard disk be attached as a local Arq source for doing an offline restore?

As for Cloudberry: I have never heard of them until today. Looks like a sloppy product.

 

[dai-leung]

The way to find out if a full restore will work is to try it and see if it works.

I have been following the discussions on getting the TB cloud backup on a hard drive delivered to one's home and how low cost B2 is. However, if the backup stored in the cloud is not identical to the original data on the Mac, HD and cost don't matter.

So far all the discussions on ARQ 5 appears to be very positive ( myself included).

I was interested in the download speed and wanted to know if putting TB backup on HD delivered to home is necessary. I came upon the first negative discussions on ARQ 5: https://www.reddit.com/r/DataHoarder/comments/68eipq/questions_about_arq_5/

I do not know if there is only ONE user (he uploaded 4TB to ACD using ARQ 5 ) who had this problem, nor do I have the technical knowledge to judge how accurate his description of the problem is, nor if only ACD has this problem.

My understanding (not sure is correct) of the discussions is as follows: during upload, ARQ breaks up, for example, a GB file into a 1000 threads (chunks?). Each chunk then makes a connection request to Amazon Cloud Drive for permission to upload. When requests are made too fast, ACD sometime rejects the requests. Meanwhile, ARQ does not know some requests were rejected. Larger file encounters more rejections than smaller files. This leads to those files in the cloud with rejections are not the same as the original files in the Mac. The user said because of this he lost some important files and this issue had not been resolved. Not sure why this problem did not show up during validation if he had validated his backup data.

 

[SteveJobzniak]

@dai-leung I don't think the review's theory is correct. Arq does this:

1. Split the file into smaller chunks (important; it helps reduce data storage needs since many files will consist of identical chunks, so only one copy of that chunk needs to be stored).
2. Encrypt each chunk.
3. Upload the chunk to the chosen cloud storage.
4. Wait for the reply from the server, saying "We received file X with hash 13819y38y12983".
5. Compare the hash that the server told you, to the hash of your own local chunk. Hashing is a way to verify that data is identical in both places.
6. If both match, Arq correctly marks that chunk as 100% successfully uploaded, and carries on...
7. Otherwise it retries if it failed...

What's more likely is that Amazon Cloud Drive had a bug. That service is very new and low-quality and has had a billion bugs and has been a huge headache for Arq's developer. I've seen data folders vanish and re-appear in my Amazon Cloud Drive when I had that service.

As for Backblaze: Their hashing is deeply embedded in their cloud service. They demand a hash before you even start the upload. Nothing gets stored on their servers if the chunk hashes mismatch. This guarantees that the data stored on Backblaze is 100% identical to what was uploaded. And Arq closely watches the server replies, to ensure that the data was stored, before it proceeds.

I trust Arq 100%. And I trust Backblaze 100%. I've reviewed the Backblaze storage design earlier in this thread.

You seem to be a very worried person with all your dozens of long, concerned posts over the weeks, and it's not a good way to live life. Just sign up to Arq and Backblaze and trust me. ;-) I've been an advanced programmer for 20 years and know my stuff. There isn't a project I couldn't code. I fully trust Arq's code/architecture and Backblaze's storage stability with my life's work. They are both beautifully designed internally. And Arq's storage format is open-source, which is how I was able to see how trustworthy the program is and how incredibly competent the programmer is. Everything about Arq is CPU and memory-efficient and perfectly designed. Everything about Backblaze's storage reliability and API is perfectly designed.

When Arq sends something to Backblaze, it stays there forever, perfectly intact.

Stop worrying and just get on with it. Have fun!

 

[dai-leung]

I trust Arq 100%. And I trust Backblaze 100%.

I am glad that someone so experienced like yourself came out and clarified the issue. I no doubt trust you else there should have been many more complains. (It is not a review. It is a discussion in which a user presented his problem.). My new understanding is that that user is incorrect (he might have some other problems) or perhaps ACD had problem in its acknowledgement, an acknowledgment was sent even though the data was not received.

Indeed, I worried too much. I agree, perhaps it is better just do it without asking too many questions. For me, a beginner and non technical, Cloud backup is very complex, requires high technical skill, so I am always afraid to get into a situation that is over my head. I am not sure if other non technical users face similar situation. I do understand those who post in this forum are very experienced programmers (I have learned a lot from them and their kind replies.). Perhaps, I am intruding, my apology.

 

[VideoFreek]

@dai-leungI've been an advanced programmer for 20 years and know my stuff. There isn't a project I couldn't code.

In that case, I'd like to pick your brain on a related topic, if you don't mind. I'm using ARQ to backup to B2 as well as to a local NAS share on my network. I'm struggling to write a shell script that ARQ can trigger to mount the volume prior to running backup, as it is capable of doing (would be nice if the developer would include the capability to do this easily, as CCC has). Since I know little about bash scripts, I've searched, etc., but struggle to come up with something that will

1) check if the volume is mounted, and if not...
2) mount the volume, passing a userid and password

Would be a bonus if I could figure out a way to not store the share password in plaintext, if that's possible. Any help or direction you could provide would be greatly appreciated.

 

[SteveJobzniak]

@dai-leung Glad to hear my words helped. ;-) I can understand your worries about data loss and what solution to pick, since all of this looks like a "black box" for computer novices. It's definitely difficult to judge this without deep technology knowledge.

But you can definitely fully trust Arq and Backblaze B2. The only question you should ask is whether the B2 price is right for you. You can use Backblaze B2's calculator here: https://www.backblaze.com/b2/cloud-storage-pricing.html. Set the "Initial Upload" value to the total size of the data you want to keep online. Set all three "Monthly Upload", "Monthly Delete" and "Monthly Download" fields to 0 GB (they are unrelated to storage costs). And set "Period of Time" to 1 Month. Now you will know your monthly cost to store your backup at B2. If the price feels good, go ahead! ;-)

@VideoFreek Ah, you want a script that mounts your NAS share so that the share gets backed up during Arq's process.

This script checks if /Volumes/SomeFolderName is mounted. If not, it runs the mount-command. To mount volumes, the script must run as root (sudo). So this won't work if Arq doesn't run shell scripts as root. I haven't used the scripting feature in Arq so I don't know. But here's the code anyway:

#!/usr/bin/env bash

MOUNTPOINT="/Volumes/SomeFolderName"

if ! mount | grep -q "on ${MOUNTPOINT} "; then
    # The volume isn't mounted yet, so let's mount it...
    sudo mount -t afp afp://user:password@10.1.1.10/SOMESHAREDFOLDER "${MOUNTPOINT}"
fi

As for securely storing the password, it can be done via the "security" shell command in macOS, to read a value from the macOS Keychain into a bash variable. But I've never needed use the "security" command so I don't know its syntax off-hand. You'll have to research that yourself.

And if you want the volume to be unmounted after Arq is done, you can use a separate script which says something like this:

#!/usr/bin/env bash

MOUNTPOINT="/Volumes/SomeFolderName"

if mount | grep -q "on ${MOUNTPOINT} "; then
    # The volume is mounted, so let's unmount it...
    sudo diskutil unmount "${MOUNTPOINT}"
    # ALTERNATIVE METHOD IF THE ABOVE ONE DOESN'T WORK FOR NETWORK SHARES:
    sudo umount "${MOUNTPOINT}"
fi

Good luck. I hope this gets you started!

 

[VideoFreek]

@VideoFreek Ah, you want a script that mounts your NAS share so that the share gets backed up during Arq's process.

This script checks if /Volumes/SomeFolderName is mounted. If not, it runs the mount-command. To mount volumes, the script must run as root (sudo). So this won't work if Arq doesn't run shell scripts as root. I haven't used the scripting feature in Arq so I don't know. But here's the code anyway:

#!/usr/bin/env bash

MOUNTPOINT="/Volumes/SomeFolderName"

if ! mount | grep -q "on ${MOUNTPOINT} "; then
    # The volume isn't mounted yet, so let's mount it...
    sudo mount -t afp afp://user:password@10.1.1.10/SOMESHAREDFOLDER "${MOUNTPOINT}"
fi

Good luck. I hope this gets you started!

For the love of God, I'm hopeless at scripting. I'm attempting to mount a share called "ARQbackups" on my NAS called "QNAPNAS", and this is what I'm using:

#!/usr/bin/env bash

MOUNTPOINT="/Volumes/ARQbackups"

if ! mount | grep -q "on ${MOUNTPOINT} "; then
    # The volume isn't mounted yet, so let's mount it...
    sudo mount -t smbfs smb://user:password@QNAPNAS/ARQbackups "${MOUNTPOINT}"

fi

Obviously, I put in the actual userID and password. The errors returned are:

line 5: unexpected EOF while looking for matching `"'
line 10: syntax error: unexpected end of file

Any ideas as to what I'm doing wrong?

 

[chown33]

For the love of God, I'm hopeless at scripting. I'm attempting to mount a share called "ARQbackups" on my NAS called "QNAPNAS", and this is what I'm using:

#!/usr/bin/env bash

MOUNTPOINT="/Volumes/ARQbackups"

if ! mount | grep -q "on ${MOUNTPOINT} "; then
    # The volume isn't mounted yet, so let's mount it...
    sudo mount -t smbfs smb://user:password@QNAPNAS/ARQbackups "${MOUNTPOINT}"

fi

Obviously, I put in the actual userID and password. The errors returned are:

line 5: unexpected EOF while looking for matching `"'
line 10: syntax error: unexpected end of file

Any ideas as to what I'm doing wrong?

My first guess is you have curly quotes in your shell script. For examples, see this post:
https://forums.macrumors.com/threads/xcode-invalid-character-in-source-file.2047556/#post-24617731

You should post the complete actual content you have in your file. You can put in dummies for the name and password, but everything else should be exactly as it is now. You can test whether you've got it postable by replacing your actual name and password with the dummies, then trying to run it. If you get the same shell errors, then post it at that point. Be sure to use CODE tags.

It may also help if you told us what editor you're using to edit this shell script. Some will do "smart quote" substitutions (System Preferences > Text > Smart Quotes) while others won't. Some will have a Substitutions > Smart Quotes or similar menu item (disable it if you have it). On the other hand, if you're using 'nano', 'pico', or 'vim' in a Terminal window, tell us that.

 

[SteveJobzniak]

@chown33 Good points. It looks like he's got curly quotes instead of straight quotes.

Anyone who is using TextEdit to write code needs to follow these instructions:

Press Cmd+Shift+T to switch to plaintext mode. Then go into the "Edit: Substitutions" menu and disable ALL of the "Smart"-options. If they're on they will ruin all code characters.

When finally saving the file, save it as a ".sh" file (such as "example.sh").

And lastly, open a terminal and go to the file's folder and type "chmod +x example.sh" (or whatever the filename is) to make the script executable.

@VideoFreek I see that you've figured out how to change to the "smbfs" and "smb://" for SMB (Microsoft's filesharing protocol) yourself, so you seem advanced enough to carry on from here until success. Good luck.

 

[dai-leung]

But you can definitely fully trust Arq and Backblaze B2.

Thanks for suggesting B2 and info on B2. I appreciate it and will consider using it in the future. Currently, I have a 1TB Dropbox account and used only 15GB. So will use it for ARQ for the time being.

For those ARQ users, who no longer have key codes for some applications and are interested in backing up all the applications, I came across an explanation from Pondini on why some applications can not be transfered:
http://pondini.org/OSX/TransferApps.html.
It seems that after restore, one will have a better luck to have all applications working if both the applications and Library folders are backuped in addition to the home folder.

For those users, who want to backup the entire Machintoh internal drive (not recommended by ARQ and the ARQ gurus), ARQ support gave the following instructions (for me, a computer illiterate, when in a catastrophe, prefer to have as many folders available for restore as possible. 1TB Data plan is cheap when compares to the uncertainty of data lost.):

"It's not possible to change the home folder to the entire Mac internal drive directly. Please select the destination under 'Configure Backups' and click 'Add a Folder to Backups'. Select 'Macintosh HD' from the list.

You can then select 'Macintosh HD' under 'Configure Backups' and hit 'Edit Backup Selections'. Uncheck the home folder from there if it exists as a separate backup job already."

 

[SteveJobzniak]

Sure... But it's not recommended to back up the entire hard drive with Arq. It's not even recommended to back up your /Applications folder with it. Arq is meant for critically important data: documents, photos, videos, projects, etc. And all of that exists in your Home-folder, and is covered by Arq by default (unless you've manually put the data somewhere else outside your Home folder somehow). But by adding the whole hard disk you'll add totally "useless" system-files to your backup, which will add hundreds of thousands of small and useless system-files to Arq's database, which will make the Arq database massive and slower. And you'll make all backup/validation run extremely slow since you'll force Arq to scan all those files too, even though they're useless.

In fact, it doesn't even make any *sense* to try to back up the entire system with Arq. Because Arq cannot restore any of those system files. You'll need to have a fully working and installed macOS system up and running already, to even be able to open up Arq and be able to restore any files. And at that point, macOS itself will disallow Arq from overwriting the system files that you are "trying to restore/download via Arq".

So don't add your whole hard disk to Arq. Just don't do it.

Arq's restore process works like this:

- Reinstall macOS (if you've lost it).
- Reinstall your applications.
- Install Arq.
- Connect to the backup location.
- Download your important documents, videos, etc, that you want to recover.

Your applications, your main OS files, etc, aren't important (they're totally generic and replaceable) and can easily be replaced by re-installing your OS and your applications. But if you do want a backup of your "whole system", for faster "getting things back to life" if things break, then they should be backed up in a completely different way (not with Arq). Such as with Apple Time Machine (which lets you restore all of its data to a blank disk to "reinstall your OS" exactly the way it was), or Carbon Copy Cloner (which creates a bootable exact hard disk clone of your entire disk and all apps).

 

[dai-leung]

It's not even recommended to back up your /Applications folder with it. Arq is meant for critically important data: documents, photos, videos, projects, etc. And all of that exists in your Home-folder, and is covered by Arq by default.

Your applications, your main OS files, etc, aren't important (they're totally generic and replaceable) and can easily be replaced by re-installing your OS and your applications.

In fact, it doesn't even make any *sense* to try to back up the entire system with Arq. Because Arq cannot restore any of those system files. You'll need to have a fully working and installed macOS system up and running already, to even be able to open up Arq and be able to restore any files. And at that point, macOS itself will disallow Arq from overwriting the system files that you are "trying to restore/download via Arq".

My sincere thanks! I have never thought my ignorant post which might have misled other users would get a clarification of how to use ARQ correctly by an ARQ guru. Non computer users like me will be benefitted by reading your clarification.

Have more questions, hope someone could help answer:
1) Does reinstalling Mac OS, reinstalling all applications, and finally restoring the ARQ home folder backup, make the newly restored Mac identical to the stolen Mac (assume stolen)?

2) Is a folder the largest ARQ restore unit ( e.g. the home folder)? During restore, one only needs to select the home folder backup to restore and not the individual files.

3) for users who had lost key codes or installation CDs for some applications, is there a problem of backing up the user's /Applications and /Library folders (as explained by Pondini)? The latter I understand is hidden, but can be made visible.

 

[SteveJobzniak]

I just thought this email today was funny and worth sharing.

I am storing ~129 GB at BackBlaze B2 via Arq... and it's so cheap that I'm not sure how they're even making any money from me, since credit card companies themselves charge a fixed $0.30 fee per transaction, plus a percentage of the remainder (usually 2.9%). So BackBlaze is only getting $0.30 minus a few more percent after transaction fees... Yikes.

But I am not complaining. I love the price.

 

[dai-leung]

Using ARQ, for a specific reason, I had upload the entire mackintosh hard drive (500GB) of my Mac to Dropbox. I did a test of full restore by ARQ and because I didn’t know what I was doing, it seemed that I might mess up the ARQ uploaded backup and the Mac’s HD, I panicked and aborted the full restore.

I am not a computer person and not confident of what I am about to do and hope someone can help to give advice on my new test plan given below.

On the Mac, I will first change the ARQ backup scheduler from “hourly “ to “manual” backup in order to suspend or freeze the backup process (but not entirely sure how to set backup to manual).

I will then make a bootable backup of the Mac on a portable external 1TB SSD using carbon copy cloner. The ARQ full restore test will be done on this SSD only.

I will use an old Mac with only 100 GB HD to do the test. The SSD will be external booted from this old Mac.

In this way, the original Mac will not be involved in the test and thus not affected by any mishap.

Likewise, freezing the ARQ BACKUP prevents corrupting the backups stored on the cloud.
On the booted SSD, since the ARQ backup schedule had been previous been suspended, ARQ will not do any more backup to create confusion. (In this step, do I need to reinstall ARQ to make ARQ work again on the booted SSD?)

On the booted SSD, I will then open ARQ and select “restore” , give decryption key, and restore /applications, /libraries etc, one folder at a time. Will select restore to “write over the current folder“ . After each folder has been restored, will reboot from SSD again to check if the booted SSD works just like the original Mac. The last folder to be restored will be /documents as it has over 400GB.

I will appreciate for your advice.

 

[Weaselboy]

https://store.arqbackup.com/?c=GGI6AMWMD5AITKRD

If anybody is looking to buy Arq, I got a promo email from them today they are having a Black Friday 40% off sale at this link. I'm not affiliated with Arq, but thought some of you in the thread might be interested in the deal.

 

[jameshenderson]

Hi all,

I periodically get errors such as this one below backing my Mac using Arq to B2:

Error: ~/Library/Application Support/Plex Media Server/Media/localhost/0/d6863a3fac7f4caf8f81b066a97c7b9908f7976.bundle/Contents/Indexes/index-sd.bif: error reading request: java.net.SocketTimeoutException: Read timed out​

I don't get the same one again in the subsequent backup. Does that mean the error was "fixed" on the subsequent backup?

thanks,
James.

​

 

[Weaselboy]

Looks like maybe the file is locked and Arq is not able to access it for backup? I'd send the Arq dev. a not and ask him what he thinks. He has always been quick to respond when I have had questions.

 

[jameshenderson]

@Weaselboy thanks - I did that a few weeks ago (maybe months now) and he wasn't sure himself so was just wondering if anyone else had experience with this. It actually isn't my Mac - its my sister's so have just emailed her to see if she was using Plex at the time.

 

[Weaselboy]

@Weaselboy thanks - I did that a few weeks ago (maybe months now) and he wasn't sure himself so was just wondering if anyone else had experience with this. It actually isn't my Mac - its my sister's so have just emailed her to see if she was using Plex at the time.

A long while back I had an error that looked like that because I was messing around with the Safari bookmarks plist file while Arq was running, so that is what makes me think Plex was in the middle of doing something there that locked that file.

 

[jameshenderson]

Yes, my sister has just confirmed she was using Plex at the time (downloading some videos to her iPad).

I assume (dangerous to assume?) that the next time Arq ran, it accessed and backed-up those files OK (there were n o errors on the next run). ...so I think the only thing I "lost" was the exact state of those files at that particular time (which is of no real consequence to me).

 

[11201ny]

Great thread.

My experience with Backblaze (a year or so ago) was a nightmare. What i learned in the process was that their backups aren't reliable. My beginnings with Arq were rocky, but now I see Arq + Amazon Drive as the the most reliable, & least expensive option. As a photographer who needs a lot of storage, I recommend the combo to peers.

 

[Weaselboy]

My experience with Backblaze (a year or so ago) was a nightmare.

Were you using the new Backblaze B2 service, or were you using the standard Backblaze backup service with the included app? I agree the Backblaze backup service is not that great, but I have not read about anybody having trouble with B2 and Arq.

 

[11201ny]

Were you using the new Backblaze B2 service, or were you using the standard Backblaze backup service with the included app? I agree the Backblaze backup service is not that great, but I have not read about anybody having trouble with B2 and Arq.

I wasn't using B2. It's got my interest.

 

[Weaselboy]

I wasn't using B2. It's got my interest.

I've been using it with Arq and it works quite well for my needs.

 

[11201ny]

I've been using it with Arq and it works quite well for me needs.

I have a 9TB (and growing) archive. Uploading was a serious issue with BB. Is B2 better?