Arq

[dai-leung]

Thanks.

I assume that whatever has been backuped online at that point is intact and once the ARQ unpaid user pays for the license, he can resume automatic backup as if nothing has happened.

One issue that is not clear is that after the initial selection of what to backup, say the user selected folder X, then one month later he changes his mind and wanted to backup the home folder instead. Is this allowed? What will happen to the backups for folder X?

 

[Weaselboy]

One issue that is not clear is that after the initial selection of what to backup, say the user selected folder X, then one month later he changes his mind and wanted to backup the home folder instead. Is this allowed? What will happen to the backups for folder X?

You might want to reach out to the dev on this one. I'm not sure what would happen if you UNselected folder X then selected the home folder instead. I know there is a process called "validation" that runs periodically to make sure the online backup matches what you have locally, and I'm not sure what that would do in your scenario.

 

[bwintx]

Thanks.

I assume that whatever has been backuped online at that point is intact and once the ARQ unpaid user pays for the license, he can resume automatic backup as if nothing has happened.

Oh, yes. Nothing was lost. The backup just didn't get updated to B2 (my chosen cloud storage vendor for Arq purposes) a few times, since I have it set to update once an hour; I was away from the Mac at the time so nothing had changed, anyway. Once I got home from work, opened up Arq, and saw the full message about the trial having expired, I paid for an Arq license and everything worked fine again.

 

[dai-leung]

Assume ARQ allows user to select and add folders in addition to home folder, Is it possible to select folders such that the ARQ backup is equivalent to the time machine backup? That is from time machine like to time machine equivalent. Also is there a fundamental reason why ARQ can not backup the entire Machintoh HD? I know it is not the purpose of ARQ and perhaps there is not a need for those who are technically savvy. I also know that there are super duper and carbon copy cloner that can backup the entire drive. But these are local backups not online.

I understand some applications can be restored by ARQ and some can not, as explained by weaselboy earlier. But I don't understand why time machine can restore all the user installed applications but ARQ can not.

As the cost of terabyte online backups becomes very affordable, Perhaps there is a large number of users like me, who have precious data to protect and are potential ARQ customers. We are not technical, have been relying on time machine to protect our data and are perplexed about online backups; it just seems so many issues to understand, just overwhelming. There comes ARQ which takes away the complexities, make it easy to do online backup. So I hope ARQ will help this group of users more and make it as easy to use as time machine.

 

[dai-leung]

Initial backup question.

I have an old Mac that does not have enough storage to support all my data stored in a newer Mac. It is the newer Mac that I want to do an initial backup, which is well known to take days or weeks.

An external SSD is used to clone the newer MAC then then It is further used to externally boot the older Mac. At this point, is it possible to do an ARQ backup of the SSD only? The SSD has the same contents as the newer Mac. Or just use the old Mac as the main computer and do an ARQ backup Of an external drive, the SSD. Is this possible?

After the initial backup of the SSD clone, for all subsequent backups, I will then switch back to use the newer MAC instead. Does ARQ know the source for the initial and subsequent backups are not the same, but their contents are the same? OR ARQ treats them as two totally different backups, so I will have two backups online corresponding to the two different sources used.

If ARQ does not know the source has changed, it will free up my newer Mac to do my daily work.

 

[dai-leung]

Your comments on my backup strategy will be appreciated. Note that I am not a computer person.

I have 500GB data on my Mac if I lose them, I will never recover professionally. Twenty thousands of research papers that I will not be able to find them and read them again and years of experimental results that I will not be able to do them all over again.

Following the discussions on this thread, the backup strategy I had just conceived is:
(1) backup by three different methods: time machine, Carbon Copy Cloner and ARQ
(2) backups stored at three different locations: two locally (office and home) and one online,
(3) for the two local locations, each location does two types of backup, one made by time machine and one made by CCC.

One of CCC bootable backup is on SSD, with the purpose that it could be used to boot up any Mac in time of emergency.

The redundancy and duplication are designed to recover from disasters such as stupid mistake of backing up to the wrong drive ruining its contents, mistaken source as destination, data corruption causing backing up garbage without knowing it, multiple HD failures, or unforeseen problem with the backup software, etc. The probability for these disasters to happen is not zero.

This strategy seems to satisfy the 3-2-1 requirement and seems to be easily implemented and easily executed as It does not requires much technical skill. All I need to do is to buy ARQ, CCC and four hard drives. After setup, the backups will run automatically in the background without interrupting work except each day when I start using the computer, I will need to setup which software backups to which hard drive. (It will be great if there is a way to pair the backup software and HD permanently and thus avoid the need to do it manually everyday.) Of course, I will need to remember to connect the hard drives to the Mac for backups to take place. But I have been doing that for the time machine backup anyway.

Cost:
Dropbox: I have a 1TB storage plan and is using only 15GB. Thus, online storage is free for me else is $10 per month.
ARQ: lifetime license $80 ( I have an edu discount), regular license $50
CCC: $40 ( I have an edu discount)
Time machine free
Samsung T5 1TB SSD $400
3 pieces of 2TB seagate ultra slim hard drive @$70 each, total $210 ( however, I have been using two such hard drives to do time machine backups. So actually, I only need to buy one more)

The grand total is about $700, a one time fee. Setup can be used for several years without recurring fees. $700 for the protection of one's precious data over several years does not seem excessive, as oppose to if I lose those data, I would be willing to pay almost anything to get it back.

 

[Weaselboy]

Your comments on my backup strategy will be appreciated. Note that I am not a computer person.

I think your plan described here is sound. But as for your question in the previous post about how switching drives around would impact an Arq, backup... I would run that by the dev to be sure what would happen.

 

[dai-leung]

I think your plan described here is sound. But as for your question in the previous post about how switching drives around would impact an Arq, backup... I would run that by the dev to be sure what would happen.

Thanks for OK my proposed backup plan. "run that by the dev", do u mean contacting ARQ support directly?

Just to let u know, u explain issue much clearer than the ARQ support at least in one case. I did ask ARQ support the question of whether I could back up the application folder then restore it later to a new Mac, the support answered back and pointed me to their documentation which is cryptic and I didn't understand to begin with, and was the reason why I asked support the question in the first place.

Surprised that why "switching drives around would impact an Arq, backup" does not have an answer. I would think that quite a few people still have an old Mac with them after they upgraded to a newer Mac with bigger SSD storage. 2TB HD is so cheap nowadays, a used one is even cheaper. Since the initial backups can take days, weeks or months, it seems like a good idea to copy the contents to be backed up to the HD first and connect it to the old Mac and use the setup to do the initial update. ARQ is asked to backup the external HD. The setup could then be left at home, always on , just do the initial backup undisturbed.

I read that 2TB HD has a sequential read speed of 100M bytes per sec, which is way higher than the upload speed permitted by many cloud providers. Thus, the HD read speed won't be the bottleneck.

Sending 500GB to 1TB of backup data to the cloud using the Internet appears to be a formidable and difficult task. I vaguely remember reading a story that a person, after many months still could not finish his initial backup because the cloud provider had limited his upload speed to 1-3 Mbits per sec. Thus, it seems that a dedicated setup use solely to do the initial backup undisturbed will have a better chance that the initial backup is sucessful and without error.

If this method works, spending $70 for a 2TB HD, even if it is considered as an upfront cost for successfully loading the initial backup to the cloud provider seems to be well worth it.

 

[Weaselboy]

Thanks for OK my proposed backup plan. "run that by the dev", do u mean contacting ARQ support directly?

Yes, that is what I meant. My concern is Arq periodically does a validation to make sure what is online matches what you are backing up. If that runs when some of the local data is not present (like on an external drive that is not attached), I just don't know what Arq would do.

 

[dai-leung]

Yes, that is what I meant. My concern is Arq periodically does a validation to make sure what is online matches what you are backing up. If that runs when some of the local data is not present (like on an external drive that is not attached), I just don't know what Arq would do.

Did ask ARQ support but not on validation. Not sure what will happen during validation when the source is "not there". How does validation works? Does it compare file by file for files stored on local disk vs. files stored in cloud even if it is an 1TB backup? During transmission errors could occur, especially during initial backup (and restore) of a TB of data. How these errors are discovered, fixed or prevented from happening in the first place is very interesting. Does ARQ validation assures that there is no error in the backup or restore even the data set is as large as 1TB?

However, ARQ said if the source has simply been moved to a new location, for all subsequent backups, there is a "relocate" button to tell ARQ where the source is now located. In this case, I would assume that it does not affect "validation".

About having a separate Mac to do the large initial backup, ARQ said it would be an overkill and unnecessary as the initial upload is limited by the upload speed, presumably very slow. The scanning of the local disk for backup often has to slow down to wait for the upload to catch up. Thus, I assume doing the initial backup using the primary Mac and let the backup running in the background is not a problem.

About changing what to backup later on, ARQ said no worry, there is a "detach" button, which cut off the link between (for example) an folder stored in cloud and ARQ. In all subsequent backups, This folder won't be backuped By ARQ anymore thus the folder is archived in the cloud until the "attached" button is selected. Then backup of the said folder will resume.

 

[Weaselboy]

Does ARQ validation assures that there is no error in the backup or restore even the data set is as large as 1TB?

Yes... that is the idea. To verify what you have uploaded matches what is on the Mac.

 

[dai-leung]

Yes... that is the idea. To verify what you have uploaded matches what is on the Mac.

For completeness, I read that ARQ does a validation every 60 days.

 

[Weaselboy]

For completeness, I read that ARQ does a validation every 60 days.

Yes... the default is 60 days.

 

[dai-leung]

Yes... the default is 60 days.

View attachment 716169

Does validation uses a lot of CPU time? And how long will it take to validate a TB?

It seems that it will be reassuring to do a validation right after a TB of initial upload.

Forgive my ignorance, how do I access the schedule page which sets the validation schedule.

 

[Weaselboy]

Does validation uses a lot of CPU time? And how long will it take to validate a TB?

It seems that it will be reassuring to do a validation right after a TB of initial upload.

Forgive my ignorance, how do I access the schedule page which sets the validation schedule.

It does use some CPU cycles, but I never notice it. It is somewhat slow. I have about 30GB I backup with Arq and it takes around 30 minutes or so to validate.

If you go to Arq preferences, then click the Edit button you will get the screen I showed you.

 

[dai-leung]

Came across some ARQ 5 information; would like to share it with first time, non technical users like me, who are trying to understand the complexity of online backup for the first time, considering using ARQ and struggling to understand ARQ and how to use it. (Source: mainly from https://www.reddit.com/r/DataHoarde...dering_arq_backup_for_acd_any_input_before_i/). Below is a summary of this info along with some personal opinions. Not sure it is correct, so please do your won research.

Reviews
Reviews by users, overall are very positive especially those who are highly technical, have more than 1TB of data, some stored their data to more than one cloud providers, cost conscious, and sophisticated users. What they like are the robustness of the product, data are encrypted before leaving the computer, able to backup multiple computers and to multiple cloud providers, simple to use and low cost. An example about simplicity, a user moved to Singapore and without any effort he was able to move his backup from Amazon US to Amazon Singapore. Discussions by these highly technical users gave the impression that ARQ is for thecnically capable users. However, my research and experience suggest it is not true, ARQ is also for non technical people, it is so simple to use even for non technical person like me. ARQ reduces the complexity of online backup to simple clicks.

Support
My personal experience about the support from ARQ is excellent. After sending support an email, I typically got answer back in 1-2 days. (CCC also provides excellent support.)

Data plan (Storage size) needed to store data in cloud -- compression ratio
In one experiment, ARQ 5 compressed a 40MB file to 16MB or a ratio of 2.5. Thus, for 1TB of data, according to time machine usage recommendation a storage size of 1TB will be sufficient.

Scanning speed
Files are first scanned then upload. In one case, a 500GB data took only a few minutes while in another case 1.5TB took 24 hrs. VMWare virtual machine bundle took a long time to scan, 2 hrs for 9GB. One user said 500GB per day, not sure if his data included VMware. During scan, if Mac is shutdown then restart, scan will start from scratch. Scan will slow down to wait for upload to catch up if far ahead of upload.

Upload speed
5 -10 MB (bytes) per sec or 18-36GB per hr , or 0.43- 0.86 TB per day. Another user uploaded 4.5TB in 2 weeks, or 0.32TB per day. Thus, the range of speed users can expect to get is 133-360GB per 10 hrs, or 300- 860GB per day. Since this includes the scanning speed, upload speed is the only number needed for initial upload time estimation.

Validation speed
Validation compares the compressed files stored in cloud against the source. The speed is 1GB per min, or 600GB per 10 hrs. (Provided by weaselboy in this thread.)

Amount of data uploaded before ARQ crashed
An user succeeded to upload 5TB before ARQ crashed.

How confident hundreds of GB to a TB of data can be safely and successfully uploaded?
A large number of users have less than a TB of data to be backuped. When I first looked into cloud backup, able to upload 500GB of data to the cloud worried and confused me the most. Forbes had a story that a user after one year still couldn't upload her 200GB+ data to a cloud provider. The reason was the provider deliberately slow down the user's upload data rate to a very slow rate. One can find similar stories on YouTube. However, I had found ther are many users who had used ARQ to sucessfully upload more than a TB of data to Amazon.

Maximum data uploaded
A user was trying to upload 22TB, an astronomical amount, to the cloud. From above, 5TB does not seems to be a problem.

Old backups, how safe?
A user left his backups in the cloud not used for several years, then accessed his files again and found them all good.

Having all your applications working after full restore
I had lost key codes for some of the applications. Thus, wish all the applications will work after restore from ARQ backup and without the need to reenter key codes. Using time machine, I was able to migrate my entire hard drive contents from a MacBook Air to MacBook Pro with all applications working.
ARQ backup only the home folder. User can obviously also backup the applications folder and /Library/Application Support/ folder (the latter was from ARQ support, some applications need the Support folder to work). However, even with these two additional folders added, there is still no guarantee that all applications will work after a full restore. The best answer is given to me by a Carbon Copy Cloner support, who said that the license registers of some apps are locked to the hard drive's UUID, or locked in some other ways. CCC does not want to break this protective locks, and as a result, these applications will not work even the user clones the Machintoh HD, then reboot the clone. If CCC backup can not make these locked applications work, then ARK backup certainly will not be able to make these applications work as well. Thus, try it with CCC bootable backup first to see if all applications work. If they do, then there is a better chance that ARQ will work too. How about those unlocked applications which are restored from ARQ backup; will they work? This question remains unanswered.

Full restore, will it work?
During my research, I have not been able to find a single ARQ user, who has several hundred GB of data stored online and has successful done a full restore. And for that matter, nor has any online backup users. Obviously the difficulty is down loading such a massive amount of data. On the contrary, time machine, CCC and Superduper have had countless sucessful full restores. Thus, the first line of defense againfst data lost catastrophe should be the local backups.

Encrypt data before sending it to the cloud, Time Machine and ransome ware
I have NDAs and proprietary research data on my Mac. Thus, ARQ encrypts data before sending it to the cloud without using a third party software is appealing to me.

Online storage is getting cheaper. A TB data plan costs only $9.99/month. A majority of users do not have 1TB of data. A large amount of unused storage space is perfect for time machine type of backup in which data are backuped every hour and each backup is saved. TIme machine like backup is a feature of ARQ. Many users are using time machine to do their backups and data of many users have been rescued by their TM backups. Thus, ARQ time machine like backup is familiar and reassuring to them. Time machine type of versioned backup has an expected benefit: some users said those backups before a ransome ware attack will not be affected unless the ransom ware instructs ARQ to delete all files, but deleting everything does not serve the purpose of the ransomer.

 

[chown33]

...
Full restore, will it work?

The way to find out if a full restore will work is to try it and see if it works.

If you're not testing your actual recovery procedures, using actual archived data, then you've left out an important part of your backup strategy. And if you don't even have an actual recovery procedure written down, in a form you can use if all your disks disappear (e.g. assume they're all stolen), then you've got an incomplete contingency plan.

In olden days, when Unix v. 6 roamed the land, we had a few DEC PDP-11's with Unix on them. They were supposed to be getting backed up to tape every night. Unfortunately, the guy responsible for doing this never bothered to verify the tapes, so when someone eventually came around looking for data to be restored, there was literally nothing there. It turns out the tape drive had a head malfunction, so it wasn't writing anything to the tape that could be read back. We had lots and lots of what were effectively blank tapes with wildly inaccurate labels on them.


I recommend doing a restore to an external disk. You should go through the entire process of formatting, partitioning, etc. exactly as you would do it by following your real procedure for a full restore. Remember, the purpose of this is to test every step in the restoration process. If you skip one, then that's an untested and unverified step.

I also recommend doing another restore trial at least once a year, again following the actual written procedure, exactly as written. You'll probably have forgotten any little tricks or caveats if you didn't write them down during the first test, so you can count on running into difficulties even if all the archived data is, in fact, valid and restorable. Take note those difficulties, formulate a solution, and verify that solution works.

 

[dai-leung]

The way to find out if a full restore will work is to try it and see if it works.

If you're not testing your actual recovery procedures, using actual archived data, ... then you've got an incomplete contingency plan.

I recommend doing a restore to an external disk...go through the entire process of formatting, partitioning, etc. exactly as you would do it by following your real procedure for a full restore.

I also recommend doing another restore trial at least once a year, again following the actual written procedure, .... you can count on running into difficulties..... Take note those difficulties, formulate a solution, and verify that solution works.

Thank you for ur suggestions. I fully agree with u : "If you're not testing your actual recovery procedures, using actual archived data of your backup, then you've got an incomplete contingency plan." Definitely will follow your advice and will do exactly what u had suggested as it makes sense.

What gives users confidence about online backup seems to be the thinking: "your backup is stored in some remote location, so when disaster hits you and you lose all other backups, the online backup will save you." For those users who have hundreds of GB or TBs of data stored online, they assume the online backup will work and can fully restore their systems, but I just couldn't find anyone who claimed he had successfully and fully restored his system using online backup. What had been done was to restore some files. But this is not the same as downloading a massive amount of data for a full restore.

I only have 500GB+ data. Have a Samsung 1TB T5 SSD and an old and unused MAC. So it is relatively easy for me to test a full restore from the ARQ backup.

 

[dimme]

I have a question about file recovery and ARQ. I plan to switch to Backblaze B2 when my Crashplan sunsets early next year. The one feature of Backblaze I really like is that they will send a hard drive out if I need to recover all 2 GB of my data. My question is If I use ARQ to upload the data vs a FTP type client is the data stored in a way where it is readable from the finder or do I need the original ARQ program and setting to recover. I thinking if my computer and external disk are stolen How do I recover to a new fresh setup?

 

[Weaselboy]

How do I recover to a new fresh setup?

No... Arq backups cannot be read by Finder and you would need to reinstall Arq and use it to recover following the instructions at this link.

https://www.arqbackup.com/documentation/pages/recovering.html

 

[VideoFreek]

The one feature of Backblaze I really like is that they will send a hard drive out if I need to recover all 2 GB of my data.

Careful here. Backblaze Personal Backup and Backblaze B2 Cloud Storage are two separate services, and the hard drive service is provided only with Backblaze Backup, which uses its own backup client and cannot be used with ARQ. Not that this is a bad thing...the Backblaze client is decent and might be the right solution for you. If you opt to go with ARQ backing up to B2, then you'll have to restore via the internet*. OTOH, 2gb is a trivial amount of data, so unless you're still on a dial-up connection, this shouldn't be a problem.


*Actually, B2 has rolled out a mail-based seeding/ restore service called "Fireball," but it involves shipping a 40TB(!) array, costs $550 + $75 shipping + a $3,000 refundable deposit, and is definitely not for home users.

 

[wombat94]

Careful here. Backblaze Personal Backup and Backblaze B2 Cloud Storage are two separate services, and the hard drive service is provided only with Backblaze Backup, which uses its own backup client and cannot be used with ARQ. Not that this is a bad thing...the Backblaze client is decent and might be the right solution for you. If you opt to go with ARQ backing up to B2, then you'll have to restore via the internet*. OTOH, 2gb is a trivial amount of data, so unless you're still on a dial-up connection, this shouldn't be a problem.


*Actually, B2 has rolled out a mail-based seeding/ restore service called "Fireball," but it involves shipping a 40TB(!) array, costs $550 + $75 shipping + a $3,000 refundable deposit, and is definitely not for home users.

On the B2 pricing page, it looks like Fireball is for seeding your initial backup, but they also now offer two options to have your data mailed to you - $89 for up to 110 GB flash drive or $189 for up to 3.5TB hard drive.

Not the cheapest, but if I am ever recovering from a complete catastrophe, not out of the realm of reasonable pricing to get the 2+ TB I would want to restore ASAP.

 

[VideoFreek]

On the B2 pricing page, it looks like Fireball is for seeding your initial backup, but they also now offer two options to have your data mailed to you - $89 for up to 110 GB flash drive or $189 for up to 3.5TB hard drive.

Oh, you're right! I missed that--thanks for the tip. For a disaster recovery situation, that would certainly speed things up. I may be mistaken, but I think you get the money back if you return the drive.

 

[SteveJobzniak]

@VideoFreek What you said about Backblaze hard drives methods didn't match my memory, so I searched. Here are the backblaze hard drive methods:

Backblaze Backup (the non-Arq solution):
- $5 per month per computer.
- Upload: Only via internet.
- Download (restore): You can pay $99 or whatever to get a drive sent to you with all your restored data, and they refund the cost if you return the drive.
- See:
https://www.pcworld.com/article/302...kup-data-for-free-if-you-return-it-after.html

Backblaze B2 Cloud Storage (the one with Arq):
- Priced per GB used.
- Upload: You can pay $550 + shipping to get a 40 terabyte disk enclosure, place your data on it, and send it back to them. It's called "Backblaze B2 Fireball". It's meant for big companies with lots of data and lots of money. However, this upload method isn't compatible with Arq. Because you can't place the Arq data on the 40 TB disk enclosure.
- Download: They can put your "storage bucket" on a hard disk and send it to you, if you contact them. Haven't seen it advertised much beyond a few twitter mentions by backblaze. But yeah it's an option for those who need it.
- See: https://www.backblaze.com/blog/backblaze-b2-cloud-storage-on-a-budget-one-year-later/

Anyway, as an Arq user you can't use hard disks for "upload", because Arq must upload the data itself to the cloud to create the proper "B2-optimized" file-structure on the cloud. However, you can possibly use hard disks for downloads/restores, if they send you the entire "bucket" with all of the necessary Arq data for Arq to recognize the backup collection. In that case, you'd add the hard drive with the backup set as a "local disk-based storage location" in Arq and hopefully it'll recognize it. But I can't vouch for that.

---

In other news:

I have now been with Backblaze B2 + Arq for a while and had a few bills from Backblaze. I'm paying about half a dollar per month to store my 105 GB. And I've been really happy with the speed both when I upload data and when I restore (download) data. I can highly recommend it. Excellent cloud storage.

 

[dimme]

In that case, you'd add the hard drive with the backup set as a "local disk-based storage location" in Arq and hopefully it'll recognize it. But I can't vouch for that.
.
I'm with CrashPlan now and one of the reasons I picked CrashPlan was that they in the past offered harddrive restore. They since dropped that service that is my reason why I an looking to switch when my subscription is up. With the time and the price of buying extra data it is worth it to me to have a hard drive shipped to me. BUT if I can recover my backup then it is worthless. Has anyone tried to restore from a hard drive with ARQ, when I spoke to Backblaze they were recommending Cloudberry I wonder if that will be more reliable.