Arq

[gilby101]

No external drive was connected to the Mac during these 48hrs for TM backup. at time t=1hr, TM tried to make the fist snapshots, and found that the size would be 500GB. Together with the original 500GB data, this snapshot would have caused the total storage to exceed the 1TB SSD capacity (or the 80% ceiling). Thus, TM did not make the first ever snapshot. As a result, at t=48 hrs, there was no snapshot on the SSD. Yes?

I am not sure what happens when there is no backup disk connected, since Time Machine can't be enabled.

You have misunderstood the nature of snapshots. A snapshot is not a copy of your data - there is no question of needing 500+500GB. It is a view of your data as it was at particular points in time. This is due to the magic of the APFS file system. It does not duplicate any data. The file system and its snapshots will consume (in your example) 500GB + 24 MB. This is sufficient to give you 24 views of the file system as it was in the past 24 hours.

And this answers q2. Yes.

As it is intended to be used, TM will also be backing up to external hard disk and after 24 hours it starts a) removing old snapshots on the system disk and b) thinning the quantity of backups on the hard disk from hourly to daily to weekly. The thinning is what Apple is referring to when talking about the last 24 hours. When I look on my TM disk I can see backups for each of the last 24 hours, the first backup yesterday and all the preceding 30 days, and one backup every week before that.

My understanding of last successful TM backup is the last backup to hard disk.
[doublepost=1557875717][/doublepost]

Arq is not really what you want for the purposes you stated...

Just to add you your excellent reply. Two key parameters in disaster recovery are "time to recover" and what "points in time" are available.

Assuming a system disk hardware failure:

CCC, without doubt, has the fastest time to recover (a reboot and away you go), but will be more limited in the points in time available. The simple quick recovery is to the time when you last ran CCC.

TM is slower to recovery (a few hours), but has many points time available.
Arq, like TM, has lots of points in time but has a long time to recover.

For the need to recover a few files (e.g. you deleted a folder by mistake, or you want an old version of a document), the time to recover is similar for all there methods (slowest for Arq from the cloud) and easy access to multiple points in time is more important. This is where TM and Arq shine.

Personally I use TM and Arq. The only time I had trouble with TM was when the external hard dive was about to die. Like you I use different products according to the nature of the disaster I want to recover from. If my income depended on fast time to recover, I would use CCC too.

 

[dai-leung]

You could backup more than that. Really anything on your drive that you want, including the applications folders. But using Arq to backup your home folder /Users/<username> is a good place to start. It includes ~/Library which has much of your application specific data, such as Notes data, application settings, etc. You can backup your applications folder and others as well. I just wouldn't think of using Arq as full system recovery, as there are better products for that purpose.

Thanks again for sharing ur thought and I agree that there are better backup products out there to set up a new Mac. My questions are not totally unrealistic. Suppose some one breaks into my apartment and takes away my Mac, my HD and SSD containing the Carbon copy cloner and time machine backups. I hope that I could just buy a new Mac and get back all my applications without reinstalling all applications as some of them are very old and I no longer have the needed info. I know it will take a long time, but this could be the last resort.

 

[nouveau_redneck]

Thanks again for sharing ur thought and I agree that there are better backup products out there to set up a new Mac. My questions are not totally unrealistic. Suppose some one breaks into my apartment and takes away my Mac, my HD and SSD containing the Carbon copy cloner and time machine backups. I hope that I could just buy a new Mac and get back all my applications without reinstalling all applications as some of them are very old and I no longer have the needed info. I know it will take a long time, but this could be the last resort.

Good point and example why offsite backup is so important.

My onsite CCC backups are made to a rotating set of disks, made weekly, and stored in a fire box (30 minutes at 1,500 degrees), and in turn it's stored inside a large fire safe (90 minutes at 1,200 degrees). That's not bad protection, but there are still potential events where I could lose those backups. Offsite Arq improves my chances of not losing my data. Most of it is in iCloud in some form as well.

 

[dai-leung]

You have misunderstood the nature of snapshots. A snapshot is not a copy of your data - there is no question of needing 500+500GB. It is a view of your data as it was at particular points in time. This is due to the magic of the APFS file system. It does not duplicate any data. The file system and its snapshots will consume (in your example) 500GB + 24 MB. This is sufficient to give you 24 views of the file system as it was in the past 24 hours.

Thanks again for your detailed answers. From ur explanation, I think I understand now. Unlike the external TM backup, which has to copy the 500GB data over as the first ever backup, however for the internal snapshots, since the 500GB data already there, the first ever snapshot only needs to point to that data and that is it. There is no need to make another copy.

Since snapshots only store changes within one hr between the earlier files and the current files, these changes are expected to be small. Also, there are only 24 snapshots. So I don’t understand why many users, who are experienced computer persons, reported hundreds of GBs were stolen by hidden snapshots. And they had to run codes in terminal to delete them.
[doublepost=1557894032][/doublepost]

Good point and example why offsite backup is so important.

My onsite CCC backups are made to a rotating set of disks, made weekly, and stored in a fire box (30 minutes at 1,500 degrees), and in turn it's stored inside a large fire safe (90 minutes at 1,200 degrees). That's not bad protection, but there are still potential events where I could lose those backups. Offsite improves my chances of not losing my data. Most of it is in iCloud in some form as well.

Wow! I have never known a person who takes such precaution of his backups. “iCloud”, do u sync ur document folder to iCloud? Do u find it useful?

 

[gilby101]

Suppose some one breaks into my apartment and takes away my Mac, my HD and SSD containing the Carbon copy cloner and time machine backups

From a risk management point of view you are correctly identifying risks and what you need to do to mitigate them. Just to share my fairly simple home use requirements and the mitigation measure I have taken.

1. Hardware failure. I don't need immediate restoration so don't use CCC (or similar). But I need/want to restore in a few hours. Time Machine suits me for that.

2. Need to recover particular files from the past (either due to stupid deletion or need for earlier version). Again Time Machine.

3. Failure of my backup disk. Buy another one and start again with Time Machine.

4. Multiple hardware failures (e.g. my Mac and backup disk). Serious, but unlikely, so prepared to accept slow restoration. Arq backup to the cloud mitigates this risk.

5. Fire or theft (the risk you are identifying). I will buy another Mac (insurance may help?), reinstall apps and restore data from the cloud using Arq. Again, it is serious threat but low likelihood. I am prepared to accept the nuisance (of a new/clean install) and slowness of cloud data recovery. If my livelihood depending on my Mac, I would make sure I had fast access to a complete offsite backup - this is nouveau_redneck's requirement.

6. Failure of cloud storage service (as well as numbers 4 and 5 above). Unlikely, but easy to mitigate. I use two cloud storage services with Arq.

Look at your risks (with seriousness and likelihood) and decide how to mitigate them. There is no one solution for everyone - we all need to look at our own needs.

 

[nouveau_redneck]

Look at your risks (with seriousness and likelihood) and decide how to mitigate them. There is no one solution for everyone - we all need to look at our own needs.

I couldn't agree more. No one strategy fits all.

Ten years ago my personal computer backups were fairly rudimentary. They basically consisted of occasionally copying files to an alternate disk within the same computer. That single backup representing the absolute minimum protection, but was for the most part sufficient given the small exposure to risk that my data at the time represented.

Today, I have an onsite and offsite plan that consists of encryption everywhere, with multiple redundant copies online and offline, using multiple archive/backup formats. My plan is documented in a spreadsheet, and I adhere to it. I know where my data is and know how to recover it given multiple failure scenarios. It's not really all the complex, its just having a strategy and being prepared.
[doublepost=1557897788][/doublepost]

Wow! I have never known a person who takes such precaution of his backups. “iCloud”, do u sync ur document folder to iCloud? Do u find it useful?

I do use iCloud drive desktop and documents feature, NOT using the optimize feature, and thus keeping everything locally on my system disk always. I like it a lot.

 

[dai-leung]

Some members of this thread use more than one cloud services to store their ARQ backup for redundancy. Is this practice necessary?

An Apple store tech support told me that sites like google drive, iCloud etc mirror their cloud servers, that is there is build in redundancy. Has B2, Dropbox or google drive ever crashed and lost users’ files?

 

[nouveau_redneck]

Some members of this thread use more than one cloud services to store their ARQ backup for redundancy. Is this practice necessary?

An Apple store tech support told me that sites like google drive, iCloud etc mirror their cloud servers, that is there is build in redundancy. Has B2, Dropbox or google drive ever crashed and lost users’ files?

The cloud services you mentioned are not all equal in features, so they should be looked at for their individual abilities.

iCloud drive is a data synchronization service only. There is no option to store files in iCloud independent from the state of the files on your local drive. It is not a good location for backing for that reason, as it will immediately replicate file changes and deletions. It does offer some redundancy for occurrences like drive failure, but no archival ability for point in time recovery.

The other services you mentioned should be looked at individually, but generally speaking they offer both file storage, and file syncing. File storage meaning if you put file out there, the file state does not change unless you change it. So you can control what is kept and for how long, with or without the assistance of an automated means like Arq.

For B2 specifically, I don't think they have a sync tool, so you need to supply your own tool for automation like Arq or many others, one that connects to their API.

 

[dai-leung]

The other services you mentioned should be looked at individually, but generally speaking they offer both file storage, and file syncing.

Thanks again for ur explanation. I am interested only in backup not file syncing. I currently use Dropbox for ARQ backup and lab collaboration. If dropbox’s cloud servers (provided by amazon?) has already had built in redundancy and will never lose my ARQ backup, is there any advantage gained for me to do another ARQ backup to blackblaze B2? It is not expensive about $50-60 per year. I will not be able to recover if I lose my data: many years of experimental results and 26,000 annotated journal papers collected over many years.

 

[nouveau_redneck]

Thanks again for ur explanation. I am interested only in backup not file syncing. I currently use Dropbox for ARQ backup and lab collaboration. If dropbox’s cloud servers (provided by amazon?) has already had built in redundancy and will never lose my ARQ backup, is there any advantage gained for me to do another ARQ backup to blackblaze B2? It is not expensive about $50-60 per year. I will not be able to recover if I lose my data: many years of experimental results and 26,000 annotated journal papers collected over many years.

I translate that to mean your data recovery need as critical. I would recommend having more than one backup means and location. That could be some combination of Arq going to multiple destinations, or perhaps another backup method such as a local cloning tool or Time Machine. Only you can ultimately decide risk versus effort, assessing the prospect of losing your data.

Holding backups in multiple formats and locations is always a good thing for important data. If one means fails for whatever reason, the other is available. Enterprises I've worked in would typically have multiple paths of recovery, tailored to the tiered importance of a given system. I follow that approach myself with my own data.

 

[Weaselboy]

Thanks again for ur explanation. I am interested only in backup not file syncing. I currently use Dropbox for ARQ backup and lab collaboration. If dropbox’s cloud servers (provided by amazon?) has already had built in redundancy and will never lose my ARQ backup, is there any advantage gained for me to do another ARQ backup to blackblaze B2? It is not expensive about $50-60 per year. I will not be able to recover if I lose my data: many years of experimental results and 26,000 annotated journal papers collected over many years.

Like @nouveau_redneck touched on a bit earlier, services like Dropbox and iCloud Drive are more a sync service than a backup. For example, if you accidentally delete three files from the Dropbox iOS app, those files will be removed from Dropbox, then also automatically removed from your Mac when Dropbox syncs next time. For this reason, you should have some other setup for your backups.

Here is what I use... I have a Time Machine disk always attached that does hourly backups. Once a week or so I update a CCC clone to another drive. I use iCloud Drive for sync, then finally Arq to B2 servers for an online backup that runs once a day. This way I have multiple backups if one of them fails or is inaccessible for some reason.

 

[dai-leung]

Like @nouveau_redneck Dropbox and iCloud Drive are more a sync service than a backup.

Here is what I use... I have a Time Machine disk always attached that does hourly backups. Once a week or so I update a CCC clone to another drive. I use iCloud Drive for sync, then finally Arq to B2 servers for an online backup that runs once a day. This way I have multiple backups if one of them fails or is inaccessible for some reason.

Thanks for answering my wwiestions, much appreciated! Now I know the difference between sync and backup cloud service. Some more questions listed below.

1) Is ur method of backup good for protecting life long work, for example, 20 years worth of photographs of an artist?

2) bringing along a hard drive to do hourly time machine backup seems to be cumbersome. Can the TM hourly backup be replaced by the TM local hourly snapshots as well as by setting ARQ to do hourly backup? Having the snapshots and ARQ hourly backup, I will then do time machine backup once a day. Is this method offers the same protection as what u are doing?

3) u don't worry if the TM hard drive crashes or B2 crashes?

 

[Weaselboy]

1. I am comfortable everything is safe with the method I'm using. I have literally a lifetime of family photos scanned onto my Mac that relies on this backup process.

2. I don't drag the TM hard drive along if I am away a few days on the MacBook. I just rely on the Arq backups during that time.

3. Not really... since I still have multiple fallback backups if one of those fails.

 

[dai-leung]

Just an observation. To back up the Mac’s precious data, it seems that quite a few members use Time Machine, Carbon Copy Cloner and ARQ, each has its unique and non-overlapping strength. Together, they seem to cover all disaster or failure scenarios. There are two camps, one with redundancy and one without.

Those who advocate redundancy do Time Machine (possibly CCC as well) backups on more than one hard drives stored at different locations and ARQ to more than one cloud providers. This is to ensure that they always have an error free backup in each product type. And obviously, for those who advocate simplicity, TM backup will be on just one hard drive only and ARQ to just one cloud. The simplicity camp, despite its smaller effort, does not seem has ever run into trouble.

 

[dai-leung]

I am confused and don’t know how to resume ARQ backup. If u could help by sharing your knowledge, I would greatly appreciated.

Just bought a new Mac with Mojave and put ARQ on manual backup mode. Using time machine backup, I migrated content of an old Mac with El Capita to the new Mac. Before migration, I had been backing up the old Mac using ARQ. However, instead of just backing up the home folder as recommended by ARQ and some experienced members of this thread , I backuped the entire old Mac. The reason was I had lost installation info of some of the applications and the other reason was just ignorant. I was thinking that if I backuped the whole Mac, after a disaster I would be able to get back all the applications or at least a majority of them.

When I was using time machine to set up the new Mac, it said it was transferring 3 things: “Applications, Documents and Data, Computer & Network Settings.” After a disaster, able to restore these 3 things is my goal. For example, the System folder is not included in these 3 things therefore I don’t need it.

Now, I would like to resume ARQ backup of the new computer. I understand from Weaselboy that for the home folder, I could just continue the ARQ backup as if nothing had happened. This is of great help because my home folder has more than 400GB of data. But because I had backuped the entire Mac, I am confused as what to do with other folders. Below are my questions.

Q1) I understand that Mac has five folders: System, Library, Users, Applications, Home.
What are the folders that contain “Applications, Documents and Data, Computer & Network Settings.”?

Q2) Are these migration essential folders different between El Captan and Mojave?

Q3) Following the home folder reasoning, for each of these folders If the difference is minor, can I simply resume the ARQ backup? Or better to delete them and start from scratch?

Q4) Do I need to use “use this backup set” to tell ARQ that I had switch computers?

 

[dai-leung]

Came across this statement on https://support.apple.com/en-us/HT201250 (Back up your Mac with Time Machine):

• “In OS X Lion v10.7.3 or later, you can start up from your Time Machine disk, if necessary. Press and hold Option as your Mac starts up. When you see the Startup Manager screen, choose “EFI Boot” as the startup disk.”

Does this mean that time machine backup is just like carbon copy cloner backup which can be used to externally boot another Mac?

 

[gilby101]

C
Does this mean that time machine backup is just like carbon copy cloner backup which can be used to externally boot another Mac?

Yes and no:
CCC backup can be used to boot and run macOS from the CCC clone disk.
TM backup can be used to boot and restore macOS to another disk.

 

[dai-leung]

Yes and no:
CCC backup can be used to boot and run macOS from the CCC clone disk.
TM backup can be used to boot and restore macOS to another disk.

Thanks! Does your answer mean using CCC backup to externally boot a Mac, I could do work as if I were using the original Mac. However using the Time Machine backup to boot another Mac , I could only “restore Mac OS (stored in Time Machine) to another disk” without the need of using Recovery mode? I could not do work for example edit an MS file?

 

[Weaselboy]

Thanks! Does your answer mean using CCC backup to externally boot a Mac, I could do work as if I were using the original Mac. However using the Time Machine backup to boot another Mac , I could only “restore Mac OS (stored in Time Machine) to another disk” without the need of using Recovery mode? I could not do work for example edit an MS file?

Yes... exactly.

 

[dai-leung]

Some members store their ARQ backup on 2 or more clouds from different providers for redundancy. It raises an interesting issue of how likely a cloud provider will lose a subscriber’s data.

I came across this interesting article https://www.backblaze.com/blog/cloud-storage-durability/. It turns out that instead of using mirroring or RAID configuration, Blackblaze uses error correcting code (Reed-Solomon Erasure Code) to store the customers’ data.

To my elementary understanding, in this method, extra bits are inserted into a block of data to allow retrieval of the correct data even if some of the bits in the block could be in error.
From blackblaze: “When you send us a file or object, it is actually broken up into 20 pieces (“shards”). The shards overlap so that the original file can be reconstructed from any combination of any 17 of the original 20 pieces. We then store those pieces on different drives that sit in different physical places (we call those 20 drives a “tome”) to minimize the possibility of data loss. When one drive fails, we have processes in place to “rebuild” the data for that drive. So, to lose a file, we have to have four drives fail before we had a chance to rebuild the first one.”

So how safe the data is?
Blackblaze uses spinning hard disks, for example 4TB HDs from Seagate. Each HD has a probability of failure.
From the above, if a file (say 1TB) is broken up into blocks of 20 bits and each bit is stored in one of the 20 HDs, then the file will be in error if 4 or more HDs fails at exactly the same time. Given a HD failure probability, the probability that the file will be in error seems to be incredibly small because it requires 4 or more HDs to fail at exactly the same time. It seems more likely the file is lost because of natural disaster such as the data center is destroyed by fire or lighting strike.

From blackblaze: “At the end of the day, the technical answer is “11 nines.” That’s 99.999999999%. Conceptually, if you store 1 million objects in B2 for 10 million years, you would expect to lose 1 file. There’s a higher likelihood of an asteroid destroying Earth within a million years,”

Some members also backup their files using other formats such as TIme Machine and Carbon Copy Cloner and presumably stored at different location. It seems that the probability that all three methods (Blackblaze, TM snd CCC) fail at the same time is infinitesimally small.

 

[dai-leung]

Found a report that amazon lost user data in 2011 and server crashed in 2017; also found safety info on google drive. These information is given below.

https://www.businessinsider.com/amazon-lost-data-2011-4
https://www.dailymail.co.uk/science...-partial-outage-affects-certain-websites.html

http://services.google.com/fh/files/blogs/googlesecuritysummary.pdf
Google runs its data centers using custom hardware, running a custom operating system and file system. Each of these systems has been optimized for security and performance. Since Google controls the entire hardware stack, we are able to quickly respond to any threats or weaknesses that may emerge.

• Google’s application and network architecture is designed for maximum reliability and uptime. Data is distributed across Google’s servers and data centers. If a machine fails or even an entire data center your data will still be accessible. Google owns and operates data centers around the world to keep the services you use running 24 hours a day, 7 days a week.

• Google Apps offers a 99.9% service level agreement, and in recent years, we’ve exceeded this promise; most recently, Gmail achieved 99.978% availability in 2013. Furthermore, Google Apps has no scheduled downtime or maintenance windows. Unlike most providers, we do not plan for our applications to be unavailable, even when we’re upgrading our services or maintaining our systems.

All the above information hopefully will give readers some idea of how safe their ARQ backup stored in the cloud is and based on their own safety criterion whether storing their ARQ backup to two or more cloud services is necessary.

 

[dai-leung]

Need advice on ARQ, Time Machine and CCC backup of an FileVault encrypted startup disk. Would appreciate your help.

Q1) does ARQ, TM, CCC each backup encrypted files or unencrypted files?

Q2) how to restore files or do a full recovery from the an ARQ, TM or CCC backup of an encrypted startup disk?

Q3) is “encrypt the startup disk” necessary? My Mac has a T2 security chip, which supposedly encrypts the SSD and provides other security features (which I don’t understand). If someone steal such Mac and also wants to steal the data, he will have to open the Mac remove the SSD and breaks into it. This is technically challenging. What is the probability that will happen? He would be better off to wipe it clean and use it or sell it. Am I correct?

 

[BigMcGuire]

Need advice on ARQ, Time Machine and CCC backup of an FileVault encrypted startup disk. Would appreciate your help.

Q1) does ARQ, TM, CCC each backup encrypted files or unencrypted files?

Q2) how to restore files or do a full recovery from the an ARQ, TM or CCC backup of an encrypted startup disk?

Q3) is “encrypt the startup disk” necessary? My Mac has a T2 security chip, which supposedly encrypts the SSD and provides other security features (which I don’t understand). If someone steal such Mac and also wants to steal the data, he will have to open the Mac remove the SSD and breaks into it. This is technically challenging. What is the probability that will happen? He would be better off to wipe it clean and use it or sell it. Am I correct?

Picture an encrypted disk as something you have to unlock before you get access to it. So by the time Arq is backing up anything, it is unencrypted so that Arq can deal with it. Before TM, CCC, or Arq can backup a disk, the disk password must first be put in by the operating system.

Question 2 is really answered by question 1. Before you can deal with an encrypted disk, the OS has to unlock it.

I do not have an answer for Question 3 - so I'll leave that to smarter Macrumor forum members than myself.

 

[Weaselboy]

Need advice on ARQ, Time Machine and CCC backup of an FileVault encrypted startup disk.

@BigMcGuire has you covered on the first two. Like he said, once the disk is unlocked all these apps just see files like on any other drive.

https://support.apple.com/en-us/HT208344

Here is a link that explains your third question. You should enable FileVault so your data is protected by a password. No way a thief is getting your data if you do that.

 

[dai-leung]

BigMcGuire, Weaselboy, Thank you both for the clear and concise answers!

I had spent days reading and struggling to understand FileVault and the consequence to backup, but the way you both explained it, it is so simple!