Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.

MacBH928

macrumors G3
May 17, 2008
8,738
3,895
Any one found a solution? I am looking for a FOSS app that will do incremental updates to the cloud of my local storage.

Cryptomator can do it but you have to manual send the files for backup OR send your whole HDD which means a full HDD backup each time.

Can't believe there is no solution for this. I do not trust the proprietary software stuff.
 

ChrisA

macrumors G5
Jan 5, 2006
12,918
2,170
Redondo Beach, California
Any one found a solution? I am looking for a FOSS app that will do incremental updates to the cloud of my local storage.

Cryptomator can do it but you have to manual send the files for backup OR send your whole HDD which means a full HDD backup each time.

Can't believe there is no solution for this. I do not trust the proprietary software stuff.
This is what I run on my Synolgy NAS. There are Mac, Windows and other versions too.
 

MacBH928

macrumors G3
May 17, 2008
8,738
3,895
This is what I run on my Synolgy NAS. There are Mac, Windows and other versions too.

Yeah i saw that one. A little bit advanced. Not sure how it works as it works from a browser and is not an app with an icon. Plus it says beta and the stable release is unsupported. I am not sure if I should trust beta software for backups.

also, can I browse the files and edit them in the cloud like a local file or its either full back up or full restore?
 

ChrisA

macrumors G5
Jan 5, 2006
12,918
2,170
Redondo Beach, California
Yeah i saw that one. A little bit advanced. Not sure how it works as it works from a browser and is not an app with an icon. Plus it says beta and the stable release is unsupported. I am not sure if I should trust beta software for backups.

also, can I browse the files and edit them in the cloud like a local file or its either full back up or full restore?
It is Open Source. I assume the reason you wanted Open Source was so you could read the code yourself and modify it as you see fit.

It runs as a service in the background. The web interface makes it easy to deploy to multiple computers and then you can access them all with a web browser. It has the basic features needed, increments, programmable retention, and deduplication. This is also exactly what Time Machine does with local storage. You can look at the backed data using the browser interface. Of course, you can't see the version-controlled, compressed, and encrypted data on the remote drive. That is the entire point. It looks like random gibberish to anyone who would look.

But really, unless you are able to audit the source code yourself and built an executable from that source code yourself there s no reason to "trust" open source.

Backblaze does exactly what you need to be done, but it is not open source and it is very easy to use and does not require any software skills.
BackblazeBackblazehttps://www.backblaze.com
 

MacBH928

macrumors G3
May 17, 2008
8,738
3,895

ehh.. same thing I do not trust what is happening in the background software. Plus it doesn't do incremental backups of the HDD.

It is Open Source. I assume the reason you wanted Open Source was so you could read the code yourself and modify it as you see fit.

Oh no. I use FOSS because I trust nothing funny is going on in the background otherwise I might as well use Google Drive.

Backblaze does exactly what you need to be done, but it is not open source and it is very easy to use and does not require any software skills.
BackblazeBackblazehttps://www.backblaze.com

same, I do not trust proprietary software with such information. Next thing you know your family pictures leaked online along your bank information
 

svenmany

macrumors demi-god
Jun 19, 2011
2,275
1,527
But really, unless you are able to audit the source code yourself and built an executable from that source code yourself there s no reason to "trust" open source.

This can't be emphasized enough.

I like what https://hungvu.tech/security-and-privacy-with-open-source-is-it-a-fallacy had to say:

As a normal user, do you have enough resources and capabilities to review the codebase? If the answer is no, then how is it different than using closed-source software? Everything is effectively abstracted away, and you are just trusting that software maintainers are doing the job right. With that said, there are a few things to consider.

  1. Are there many active (and experienced) contributors, so the vulnerabilities can be detected promptly?
  2. Are the maintainers themselves well-versed in cybersecurity?
  3. Is the project under active development?
  4. Does it have a reasonable vulnerability disclosure policy and a good track record?

With respect to the topic of this thread, I would trust the proprietary Backblaze much more than I would a solution from a small group of not well-known open source developers. If Backblaze suffered a breach, the whole world would hear about it. If a small open source project suffered a breach, you might never hear about it.
 

ChrisA

macrumors G5
Jan 5, 2006
12,918
2,170
Redondo Beach, California
ehh.. same thing I do not trust what is happening in the background software. Plus it doesn't do incremental backups of the HDD.



Oh no. I use FOSS because I trust nothing funny is going on in the background otherwise I might as well use Google Drive.



same, I do not trust proprietary software with such information. Next thing you know your family pictures leaked online along your bank information
If you personally can't read the code and don't compile the source code then how do you know
  1. there is no intentional back door in the code and
  2. there is no error that creates an unintentional back door in the code and
  3. that the executable you are running have anything at all to do with the source code you inspected

It is actually more reasonable to trust a commercial system like Backblaze than some random Open Source project that uses technology (like software) that you don't understand and can't read and compile.

Also, if your data is encrypted at your end and then moved to the cloud. Then it does not matter who can see it. You may as well give the entire world read-access. As long as no one but you has the key. But this is the weak point. How do YOU store a 256-bit key? Many times it is generated from a password when ever it is needed but You better read that OS code to be sure the random unknown to you amateur programmer got the details right and did not make a mistake.

Then you say "Don't trust what is going on in the background".. Then turn off the computer. Try this experiment on your Mac. Type the command "ps -ax | wc -l" in a terminal window. I do this on my new M2 powered Mac and the command prints "432" to the screen. Your number may be different. This is the number of "background tasks" running on my computer. You should expect to see "hundreds" of them. This is the way your Mac works. it always runs hundreds of these tasks.
 

Arctic Moose

macrumors 68000
Jun 22, 2017
1,599
2,133
Gothenburg, Sweden
You could try an encrypted disk image. If you make it a sparse image, then it will only be as large is necessary. Do you know about that?

IMHO, this by far the cheapest and most secure for the money! And it's Time Machine backed up!

I have been doing exactly this for years and years, it has saved me lots of times. I put a copy of it on a web server every now and then too, so if I am ever locked out of absolutely everything, I still only need Internet access, the correct url, my sparse image password and a computer capable of mounting a disk image to get back into my most important stuff.
 
  • Like
Reactions: KaliYoni

svenmany

macrumors demi-god
Jun 19, 2011
2,275
1,527
If the current request (an extension of the OP's request) is to find software for encrypted backup, but use no proprietary software, then the first step has to be to give up macOS. That proprietary software has the most intimate access to all your documents. Of course, you also have to give up any commercial productivity tools that you might use to access your documents. Actually, you have to make sure no commercial software has access to your disks at all.

Linux, and any well-known distribution of it, is open source software that I would trust. I would tend to trust backup software installed by default in such a distribution. I see that Ubuntu's website mentions Bacula and rsnapshot. Even if those don't support encryption, one could mount an encrypted drive (on a Synology NAS, for example) and use that as a backup destination.

I don't see any other avenue for someone who doesn't want to use proprietary software but still wants something user-friendly.
 
  • Like
Reactions: BigMcGuire

ChrisA

macrumors G5
Jan 5, 2006
12,918
2,170
Redondo Beach, California
...can I browse the files and edit them in the cloud like a local file or its either full back up or full restore?
This is 100% impossible if the remote data is encrypted. You simply can not have it both ways. If the crypto key never leaves your computer then you MUST move the data back to your computer before it is readable.

But if you allow the key to move to the cloud server then the server can show you the data.

You also have a problem using a Mac. The key is stored on the Mac but how do you know Apple does not spy on your computer and copy the key? Even if you trust Apple, ow do you know they did not make an error that allows some third party to copy your keys

So you decide the hard-build a computer from basic chips. You design the circuit boards and write all the software yourself. Even if you are very carful how can you know that you did not make an error? You might be transmitting you secret data to the world and not know it.

OK, there is a way... I used to work in an industry that required Iron-Clad security. One basic rule was that no computer can be attached, even very indirectly to the Internet. The entire facility was inside an electromagnetic-shielded room. And the rule was that data was on removable storage and was to be locked in a safe if it was out of sight. We would sign for the data, use it then put it back. Cell phones and the like were not allowed in the rooom. Guards watched the safes at night.
 
  • Like
Reactions: HDFan

JamesMay82

macrumors 65816
Oct 12, 2009
1,474
1,205
You could try an encrypted disk image. If you make it a sparse image, then it will only be as large is necessary. Do you know about that?

I just did a test; it works fine with iCloud Drive. That is, I created an encrypted sparse bundle disk image on one machine and put some files in it. Then I went to another machine and mounted it; the files were there.

I might be remembering it wrong but the original time capsule use to create a sparse bundle but i found that over Time they would be prone to corruption/errors and not moment. It happen twice to me.

i understand the ops paranoia but if you are so paranoid why trust the cloud at all. I wouldn’t and i would just stick to local back ups Yourself. Using the old faithful 321 method.

I’ve just had to do a restore from icloud of our photos and it took absolutely ages and in the end i just restored from a local back up and created a new library.

I just watched this on you tube from snazzy labs..

he makes a good point about using blu ray discs for your long term storage. I.e. a burglary wouldnt be robbing your cds
 

MacBH928

macrumors G3
May 17, 2008
8,738
3,895
This can't be emphasized enough.

I like what https://hungvu.tech/security-and-privacy-with-open-source-is-it-a-fallacy had to say:



With respect to the topic of this thread, I would trust the proprietary Backblaze much more than I would a solution from a small group of not well-known open source developers. If Backblaze suffered a breach, the whole world would hear about it. If a small open source project suffered a breach, you might never hear about it.

If you personally can't read the code and don't compile the source code then how do you know
  1. there is no intentional back door in the code and
  2. there is no error that creates an unintentional back door in the code and
  3. that the executable you are running have anything at all to do with the source code you inspected

It is actually more reasonable to trust a commercial system like Backblaze than some random Open Source project that uses technology (like software) that you don't understand and can't read and compile.

Also, if your data is encrypted at your end and then moved to the cloud. Then it does not matter who can see it. You may as well give the entire world read-access. As long as no one but you has the key. But this is the weak point. How do YOU store a 256-bit key? Many times it is generated from a password when ever it is needed but You better read that OS code to be sure the random unknown to you amateur programmer got the details right and did not make a mistake.

Then you say "Don't trust what is going on in the background".. Then turn off the computer. Try this experiment on your Mac. Type the command "ps -ax | wc -l" in a terminal window. I do this on my new M2 powered Mac and the command prints "432" to the screen. Your number may be different. This is the number of "background tasks" running on my computer. You should expect to see "hundreds" of them. This is the way your Mac works. it always runs hundreds of these tasks.

its a matter of who you trust. FOSS might have issues, but its not like proprietary is guaranteed to be secure. Then there is the privacy issue. I pick and choose true and tested foss apps made by a reputable group. For example I trust to use Firefox (FOSS) which has like 5% market share of internet users (50 million users?) made by Mozilla. I won't be doing my banking on Pale Moon.

I can't read the code, no, but I trust the "community" out there. When Debian(FOSS) is used by programmers, when it has been in development for like 30 years, when Ubuntu basis its enterprise Linux distro on it, when Linode, Siemens, and the governments of Canada and USA trust Debian...I do not need to read the code myself [Source]

@svenmany You are correct about MacOS. Its a reason why I trust to keep my software on Mac not Windows. I trust Apple not snooping on my local files but I do not trust them when they are on their iCloud servers. If I see more movements from Apple towards things like this and this , I will seriously consider switching to Linux full time.
 

MacBH928

macrumors G3
May 17, 2008
8,738
3,895
This is 100% impossible if the remote data is encrypted. You simply can not have it both ways. If the crypto key never leaves your computer then you MUST move the data back to your computer before it is readable.

But if you allow the key to move to the cloud server then the server can show you the data.

Ah , I see now why I can't find a solution to my problem. I was hoping to treat the cloud storage like a local USB drive that is encrypted by filevault.

So how does Blackblaze work? you have to redownload your whole data back to regain a specific file? how about Cryptomator? it unlocks the files and you can browse them but they are not on your disk they are in the cloud

You also have a problem using a Mac. The key is stored on the Mac but how do you know Apple does not spy on your computer and copy the key? Even if you trust Apple, ow do you know they did not make an error that allows some third party to copy your keys

I don't. So far I am keeping the trust

So you decide the hard-build a computer from basic chips. You design the circuit boards and write all the software yourself. Even if you are very carful how can you know that you did not make an error? You might be transmitting you secret data to the world and not know it.

I am not that paranoid, but I am also not dumb enough to store my data on Facebook servers. But this is actually a problem. FSF doesn't use anything that has proprietary software and AFAIK some circuit board chips like intel processors has firmware that is proprietary close source. I do not know what the FSF or Richard Stallman use for their own computers or as a cellphone.

In a perfect world, I would hope all this software would be open sourced even if it was paid or not free to fork and monetize. At least we can trust it.
 

MacBH928

macrumors G3
May 17, 2008
8,738
3,895
This is something I was looking into recently. I use Veracrypt to encrypt local drives but I wanted something I could use for cloud backups and Veracrypt is no good because even a 1 byte change to a 1tb container would require the whole 1tb to be reuploaded.
I've been recommended instead to use a product called cryptomator. It's open source and audited just like Veracrypt but it's designed for encrypting things on the cloud.

Edit: Just after posting I re-read your initial post and this time spotted it in your list that you'd rejected. Oh well

I did not reject Cryptomator. The problem is it does not do incremental backups. Lets assume you have a 10GB "Photos" folder. You add 2 images to it that are 10MB. If you upload the "Photos" folder to Cryptomator, it will reupload the 10GB of data. Now imagine this on your whole HDD.

I heard you can use it as a destination for Carbon Copy Cloner (does incremental backups) but I have not tested it.

I might be remembering it wrong but the original time capsule use to create a sparse bundle but i found that over Time they would be prone to corruption/errors and not moment. It happen twice to me.

Are you sure the corruption is because of the sparse bundle not Time Machine. I once used Time Machine to do backup and it failed me on the first attempted on restoration. Paid for Carbon Copy Cloner, I never had any problems.

i understand the ops paranoia but if you are so paranoid why trust the cloud at all. I wouldn’t and i would just stick to local back ups Yourself. Using the old faithful 321 method.

yes I am aiming for that but wanted to add the cloud to my "321" method. I won't lie, I do have paranoia about drives failing. Its not helping that one of my backup drives suddenly failed to mount hence the current thread of trying to keep a cloud backup.

I’ve just had to do a restore from icloud of our photos and it took absolutely ages and in the end i just restored from a local back up and created a new library.

Are they really that slow? I thought internet is fast now with 100s Mbps connection. If you got 100mbps connection you should be able to download 36GB of data in an hour...in theory.

I just watched this on you tube from snazzy labs..

he makes a good point about using blu ray discs for your long term storage. I.e. a burglary wouldnt be robbing your cds

He is not wrong. Discs will last for decades (I know I had decades old discs) and they are not prone to failure to failed chips, data corruption, reading needle malfunction, or viruses. Two problems though:-

-BD discs has limit of about 100GB per disc.

-A guy on the internet told me this. In the future you will have problem with interfacing with that media. You won't find disc drive to read from or the OS does not have drivers to read from them. Can't argue with him as this is true with floppy discs and zip drives. BD discs and drives are already on the scarce side of things in stores and nearly no computer comes shipped with a built in BD drive. Even videogame consoles are releasing drive-less units.
 

Arctic Moose

macrumors 68000
Jun 22, 2017
1,599
2,133
Gothenburg, Sweden
This is 100% impossible if the remote data is encrypted.

While technically true, using a sparse image and iCloud Drive as described above the end result is in practice exactly what has been requested.

The files are stored locally, and only bands that have changed are synced back to the cloud when the image is unmounted.

EDIT: Sorry, I see now that the question was about accessing/editing remotely without downloading/restoring.
 
Last edited:

svenmany

macrumors demi-god
Jun 19, 2011
2,275
1,527
Ah , I see now why I can't find a solution to my problem. I was hoping to treat the cloud storage like a local USB drive that is encrypted by filevault.

Even with client-side encryption only, I can see a way to accomplish this (at a very high level). So, it's not logically impossible and it could be something is available that does it.

I think what would come close to satisfying your requirements is a dedicated, open source application that

- reads encrypted data from a remote location (of just the single file you're interested in)

- decrypts the data locally with encryption keys that only you have

- keeps that decrypted data in RAM, in some in-memory filesystem (like a RAM disk)

- supplies file handles into that filesystem to the standard programs you use (e.g. Preview, TextEdit)

If you were to open a file in some application from a USB drive, quite likely the entire file would effectively be loaded into RAM as you navigate around looking at it. The above approach would be more aggressive, getting the whole file loaded into RAM at the start. I guess the file would be in RAM twice, once in the RAM disk and once in the working memory of the program you're using. But, it's just one file at a time, so probably not a big deal.

The devil is in the details. And, more experienced people would think of betters ways to do it.
 

MacBH928

macrumors G3
May 17, 2008
8,738
3,895
Even with client-side encryption only, I can see a way to accomplish this (at a very high level). So, it's not logically impossible and it could be something is available that does it.

I think what would come close to satisfying your requirements is a dedicated, open source application that

- reads encrypted data from a remote location (of just the single file you're interested in)

- decrypts the data locally with encryption keys that only you have

- keeps that decrypted data in RAM, in some in-memory filesystem (like a RAM disk)

- supplies file handles into that filesystem to the standard programs you use (e.g. Preview, TextEdit)

If you were to open a file in some application from a USB drive, quite likely the entire file would effectively be loaded into RAM as you navigate around looking at it. The above approach would be more aggressive, getting the whole file loaded into RAM at the start. I guess the file would be in RAM twice, once in the RAM disk and once in the working memory of the program you're using. But, it's just one file at a time, so probably not a big deal.

The devil is in the details. And, more experienced people would think of betters ways to do it.

I was hoping to use the cloud drive like a local USB drive+FileVault or veracrypt.

I am still wondering how does Cryptomator can unlock the files while they are still in the cloud? Also iCloud, BackBlaze, Filen.io , proton drive claim that its e2e encrypted but the files are unlocked in the cloud not my system. I am guess the server software has to work with the local software to do this. Using 3rd party app with the cloud storage wont work. Correct me if I am wrong.

@svenmany I think I read somewhere you use Arq backup. Whats your opinion on that? any issues? It says it does e2e. Can your browse your files in the cloud?
 
Last edited:

MacBH928

macrumors G3
May 17, 2008
8,738
3,895
While technically true, using a sparse image and iCloud Drive as described above the end result is in practice exactly what has been requested.

The files are stored locally, and only bands that have changed are synced back to the cloud when the image is unmounted.

EDIT: Sorry, I see now that the question was about accessing/editing remotely without downloading/restoring.

Yeah, the idea is to have a backup of my HDD in the cloud. Your method will cause me to use double storage on my HDD. One is my Data , the second is the sparse bundle. But this could be a work around solution by just increase local storage space to double.

I might give it a thought, as long as incremental backup is an option I am ok. Uploading my whole drive data each time is not efficient.
 

MacBH928

macrumors G3
May 17, 2008
8,738
3,895
I think I might have finally found the answer. Renting a VPS and installing NextCloud on it. Its open source and does e2e.
 

svenmany

macrumors demi-god
Jun 19, 2011
2,275
1,527
I think I read somewhere you use Arq backup. Whats your opinion on that? any issues? It says it does e2e. Can your browse your files in the cloud?

I do use Arq. I bought a lifetime license years ago. It's absolutely great. You can browse the full backup history in the app without downloading to your disk. The backups are encrypted in the cloud and client-side encryption is used. I backup to Dropbox, OneDrive, and AWS. Arq supports AWS in implementing a ransomware protection feature. https://www.arqbackup.com/blog/immutable-backup-ransomware-protection/. However, to open any particular file, you have to restore it to your computer.

I am still wondering how does Cryptomator can unlock the files while they are still in the cloud? Also iCloud, BackBlaze, Filen.io , proton drive claim that its e2e encrypted but the files are unlocked in the cloud not my system.

For the tools that say they are e2e encrypted (a misnomer), the files are not unlocked in the cloud. Inside your browser is a full application running in Javascript. That application uses cryptography libraries in the same way a desktop application uses them. It has access to your unlocked secret key just as a desktop application would.

Even though it appears to you as if the browser is showing you unlocked resources on a server, it's just not the case. The encrypted data is transferred to your browser, decrypted, and displayed to you.

Regarding the e2e misnomer, wikipedia says

Some encrypted backup and file sharing services provide client-side encryption. The encryption they offer is here not referred to as end-to-end encryption, because the services are not meant for sharing messages between users. However, the term "end-to-end encryption" is sometimes incorrectly used to describe client-side encryption.


I think I might have finally found the answer. Renting a VPS and installing NextCloud on it. Its open source and does e2e.

Can you open a remote file in a desktop application running on your computer without downloading the file to your computer? That's what you said you wanted when you gave the analogy of an attached USB drive.
 

MacBH928

macrumors G3
May 17, 2008
8,738
3,895
I do use Arq. I bought a lifetime license years ago. It's absolutely great. You can browse the full backup history in the app without downloading to your disk. The backups are encrypted in the cloud and client-side encryption is used. I backup to Dropbox, OneDrive, and AWS. Arq supports AWS in implementing a ransomware protection feature. https://www.arqbackup.com/blog/immutable-backup-ransomware-protection/. However, to open any particular file, you have to restore it to your computer.

Can you edit/re-organize the files and folders? Like add new folder, rename, or send a file to a specific folder?

For the tools that say they are e2e encrypted (a misnomer), the files are not unlocked in the cloud.
Even though it appears to you as if the browser is showing you unlocked resources on a server, it's just not the case. The encrypted data is transferred to your browser, decrypted, and displayed to you.

Can you clear me up on this, when I access 50GB of files on an encrypted cloud service, the whole whole 50GB is transferred to my computer before I can unlock them and see them? Or just the folder/file structure while the real files remain in the cloud?

Can you open a remote file in a desktop application running on your computer without downloading the file to your computer? That's what you said you wanted when you gave the analogy of an attached USB drive.

Yeah I kind of wish for something like this but after everyone explained to me here how everything works I have to go with the next best thing. I am ok with downloading the single file I want to open, not ok with having to redownload the whole backup just to use the files.

The missing piece here is incremental backups of the local HDD. Does Arq do incremental backup of HDD?
I know its not FOSS but I am running out of solutions. I guess I have to give in some trust. I am between BackBlaze and Arq.
 

svenmany

macrumors demi-god
Jun 19, 2011
2,275
1,527
Can you edit/re-organize the files and folders? Like add new folder, rename, or send a file to a specific folder?

Arq just synchronizes your folder structure to the cloud location. You chose which folders to send up and then the entire folder structure under each base folder is copied to the cloud. You can add fancy exclusion rules under each base folder. There's no way to upload individual files to different cloud locations of your choosing.

Does Arq do incremental backup of HDD?

Yes. Only changes are uploaded. When reviewing all the different backup sets in the cloud, the files which have been modified since the last backup set are marked as having been modified. It seems similar to how Time Machine works.

Can you clear me up on this, when I access 50GB of files on an encrypted cloud service, the whole whole 50GB is transferred to my computer before I can unlock them and see them? Or just the folder/file structure while the real files remain in the cloud?

You're making me think :) I'm going to guess at how these services work because I can't imagine them working differently.

The directory structure and file names are separately encrypted. That stuff is probably downloaded all at the same time since it's so small. That allows you to browse around to figure out which files interest you. I suppose some of it could be "lazy"; that is, only when the directory tree is expanded is that portion of the structure downloaded. But still, all very small and quick to download, decrypt, and display.

Yeah I kind of wish for something like this but after everyone explained to me here how everything works I have to go with the next best thing.

I can't believe it doesn't exist (except it might not be open source).

I can't seem to let this one go and stop thinking about it. This is so achievable, I wish I had the competence and time to write something. Too much to learn in too little time.

Forget about RAM disks. I'd start by looking at macFUSE. I'd implement a file system that delivers the directory structure and file names which were decrypted from a server download. Then when a particular file is opened from within the backup app, launch services would fire up the application with a file handle into my custom filesystem which would download, decrypt, and deliver the file to the calling application. Of course, I'm already stressing over all the complications I'd encounter - installation, usability, latency. I'm not even going to do this and I'm stressing about timelines - I really gotta let it go.
 

chown33

Moderator
Staff member
Aug 9, 2009
10,999
8,887
A sea of green
I can't seem to let this one go and stop thinking about it. This is so achievable, I wish I had the competence and time to write something. Too much to learn in too little time.

Forget about RAM disks. I'd start by looking at macFUSE. I'd implement a file system that delivers the directory structure and file names which were decrypted from a server download. Then when a particular file is opened from within the backup app, launch services would fire up the application with a file handle into my custom filesystem which would download, decrypt, and deliver the file to the calling application. Of course, I'm already stressing over all the complications I'd encounter - installation, usability, latency. I'm not even going to do this and I'm stressing about timelines - I really gotta let it go.
Your description is basically what a file-syncing cloud service does. If you think of the local copy of the file as a cached version, then it's almost exactly what file-syncing cloud services do.

In a file-syncing service, the remote data on the server is structured in a hierarchical way, with files individually available. The remote files can be downloaded to a local host (e.g. a new MacBook) simply by logging into the account for the syncing service, and telling it you want the file or files. The local OS will then synchronize a local copy of the file (i.e. a locally cached version) with the master version on the remote server.

Files are created locally, i.e. in the cache, and then eventually synced to the remote service. That is, the latency of cache write-back can be considerable, but that doesn't change the fundamental nature of things.

I'd say the caching strategy for a "file-sync service" is more aggressive than one for a simple "remote file-system" service, but that's pretty much the only difference. Both services support write-back, so when the local copy is modified, it's eventually written back to the master version on the server. The server may maintain a history of different versions, so one can revert to prior versions of any file. The main visible distinction is that something described as "file-sync" is more active about keeping in sync with the remote versions of files. In caching parlance, it more aggressively pre-fetches data based on algorithms like recency, locality of reference (files in same dir), etc.

The question of encryption is orthogonal to the question of a remotely accessible file-system, just as it is for any local file-system.

The question of whether the remote files are backups or are the master versions of the files is also orthogonal to the remote file-system. In general, a backup of any file is write-mostly and read-rarely, while a typical file is read-mostly and write-occasionally, except while being actively edited. Those patterns lead to different caching strategies, but the file-system in both cases can be identical.
 

svenmany

macrumors demi-god
Jun 19, 2011
2,275
1,527
The question of encryption is orthogonal to the question of a remotely accessible file-system, just as it is for any local file-system.

In the current use case, encryption cannot be added independently. A critical part of an encrypted cloud storage is that the directory structure and file names are encrypted; nothing is revealed. The file system we require will present a directory structure that doesn't exist in the cloud. It's not the case that encryption can be optionally added. Encryption is a fundamental part of the file system which is acting as a translation layer.

Since Arq has been brought up, it might be worth mentioning how Arq stores things in the cloud. I did a sample backup of a simple directory structure.

orig.jpg

And here's what it looks like in the cloud storage:

backup.jpg

Arq backup does display the original directory structure and the file names in its application window. But, it doesn't expose a file system that can be consumed by other desktop applications.

An interesting test case, not involving encryption, is to rewrite BetterZip using macFUSE. BetterZip presents the full directory structure and file names of a zip archive without decompressing it. When you double click on a file, it launches the associated application, opening that file. But how does it do that? It expands the chosen file and saves it to a temporary directory. A file handle on that temporary file is what is given to the application.

Writing a temporary file in this way is inappropriate for encrypted content (especially if there's risk they won't be cleaned up). Also, the specific request is to be able to work with the files in the cloud without copying them to disk - as if the encrypted cloud storage was an attached drive.
 
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.