Storage ideas

[eyoungren]

So, this isn't a serious issue, just looking for any possible ideas. Between several Macs on my network I have a lot of empty storage sitting around. There are at least four or five Macs I could potentially fill drives up with just stuff.

Things I currently use my storage for:

- Downloads (my best SL Mac Mini is the target for downloads for all my Macs).
- Burn: A 6TB drive on my MacPro is used to store anything I've download and intend to burn (eventually) to disc.
- VMs: Also stored on my MacPro 6TB drive.
- Media: Stored on the NAS, backed up to the G4 PowerMac. The NAS is 6TB and the G4 backup drive is also 6TB. There's still about 3.5TBs left to fill on both drives.
- Tools, long-term storage, OS, frequently used app installs: Also stored on the MP 6TB drive.
- Graphics: Image library, design work, wallpaper images, graphic work. That's on the 2TB RAID and I still have 1.95TB left to fill.
- Scratch/backup: for Photoshop and InDesign/QuarkXPress.

Some other notes. One drive on the MP is 2TB (I have four drives). It's solely being used to store images I take off the internet that I'm interested in but don't rise to the level of graphics I'd use for desktop wallpaper. Drive 4 on the MP is 6TB, 4TB of which is intended for Dropbox. That leaves 2TB for other stuff. Same thing with an external enclosure for the L:09 Mac Mini.

Finally, I have a bunch of 1 to 3TB drives that are just sitting around. I figure those will become backup drives for the stuff I intended to burn, but if anyone has a better use for them I'm listening.

Again, pretty much any suggestion on what to do with my space is valid and welcome. I mean, come on. I'm using one 2TB drive to essentially store junk images!

 

[Amethyst1]

Again, pretty much any suggestion on what to do with my space is valid and welcome.

• Archive your favourite YouTube content
• Collect ISOs of every version of every BSD and Linux distro out there
• Create more VMs
• Keep rips of all your CDs, DVDs and Blu-rays
• Mirror this
• Sign up for an online PVR service (a friend of mine has collected dozens of TBs worth of movies that way)
• Store an offline version of the entire Wikipedia in every language

 

[rampancy]

Being the digital packrat that I am, I'm always using my spare HDDs to archive my DRM-free media purchases (e.g. from Humble Bundle, StoryBundle, etc.), especially since some of them came from time-limited download links -- and on that note, I'm way behind in archiving my GOG purchases.

 

[Wowfunhappy]

Use them all as Time Machine drives so you'll have additional backups should something fail.

That's not a "well I didn't have anything better to do" use case, it's important!

 

[TheShortTimer]

Use them all as Time Machine drives so you'll have additional backups should something fail.

That's not a "well I didn't have anything better to do" use case, it's important!

Why Time Machine and not SuperDuper! or CCC? Is there a particular preference with TM?

Just curious.

 

[Wowfunhappy]

Why Time Machine and not SuperDuper! or CCC? Is there a particular preference with TM?

Just curious.

No good reason. I do generally try to utilize what's built into the OS before reaching for a third party solution.

I make use of the Time Machine interface itself regularly frequently to see revision histories. It also makes incremental backups relatively easy, even to multiple backup backup disks which I don't keep connected all the time, giving me reminders every ten days to plug on one or the other disk for a backup.

This last piece is important to me as a ransomware prevention strategy--I always want one relatively recent backup in cold storage!

 

[TheShortTimer]

No good reason. I do generally try to utilize what's built into the OS before reaching for a third party solution.

I make use of the Time Machine interface itself regularly frequently to see revision histories. It also makes incremental backups relatively easy, even to multiple backup backup disks which I don't keep connected all the time, giving me reminders every ten days to plug on one or the other disk for a backup.

Cool. I was just wondering whether you'd found any benefits with using Time Machine as I've never utilised it.

This last piece is important to me as a ransomware prevention strategy--I always want one relatively recent backup in cold storage!

Earlier this week I received a generic ransomware e-mail. I was mildly amused at the title "I RECORDED YOU!" because my iSight cameras have always been covered.

 

[Wowfunhappy]

Cool. I just wondering whether you'd found any benefits with using Time Machine as I've never utilised it.

Yeah, as I said having Show Revision History as an option in most Cocoa apps is really great, that relies on Time Machine.

 

[eyoungren]

Use them all as Time Machine drives so you'll have additional backups should something fail.

That's not a "well I didn't have anything better to do" use case, it's important!

I forgot about that…

Each Mac has a DAILY backup to the NAS and a WEEKLY backup to my Mac Pro. The weekly backup goes to the Dropbox folder, which puts it on Dropbox.

Those backups are all Carbon Copy Cloner.

 

[Ifti]

Why Time Machine and not SuperDuper! or CCC? Is there a particular preference with TM?

Just curious.

Use both!!
I use TM for both my systems (to separate drives) quite regularly (I run a manual backup as and when I need to rather than automatic).
I also create a CCC backup onto a separate drive every month or two.

 

[eyoungren]

Cool. I was just wondering whether you'd found any benefits with using Time Machine as I've never utilised it.

I discovered a few negatives (for me) after long use.

1. You can't just move the backups. 2. If there is a problem with the backup, TM takes an extraordinary length of time to fix it - if it can. 3. The diskimages tend to go corrupt or cannot be opened. Trying to delete them or move them becomes a nightmare. 4. Depending on type of Mac, you get either a sparse image or sparsebundle - you can't control which. 5. Between PowerPC and Intel and even versions of Intel OS you might have different methods of establishing the TM backup. 2006 Intel Mac TM backups from SL are a nightmare to get going AND keep going for instance. 6. Sparse bundle tends to go bad faster than sparse image on a network drive. 7. You cannot control when the backups happen and establishing a rotation is something I never figured out because of all the other problems. 8. Determining whether Time Machine uses AFP or SMB based on the Mac/OS is frustrating. 9. You can control which DRIVE a TM backup goes to - but NOT a folder on that drive!

Basically, I had too many problems with Time Machine. Using CCC allowed me to do as I mentioned above, use spareimages (as opposed to sparsebundle) and control when and where the backups occur. Outside of CCC I can open these images as I need them and moving them isn't a problem. CCC is set to only copy or delete files that do not exist on the source so each backup is essentially incremental. And unlike TM backups I don't have 'multiple' incremental backups by date. It's the entire drive or selected items.

Finally, CCC works from my Tiger G3 to my 2009 Minis and Mac Pro. While the app interface is different OS to OS, I can still control things the same way.

I lost a lot of TM backups because they went bad mysteriously or corrupted because I moved them. Sparse bundles in particular are a pain to move and NAS drives or network shared drives (particularly those shared via SMB) tend not to like them.

EDIT: The lone PC I have uses an entirely different third party software to create weekly backups directly to my Mac Pro's Dropbox folder.

EDIT2: Referring to say a PowerPC TM backup and a 2006 Intel Mac backup - The PowerPC backup to a network drive/share uses AFP. Creating the backup will fail. You have to create the disk image first (name it right), move it to the drive/share and then restart the TM backup. Same with the 2006 Intel TM backup. Only, you need to edit the network drive/share so that TM sees it as a backup drive. And you have to use an SMB connection, not AFP. It's problems like these that I just became frustrated and angry over, especially when they failed (or only worked once).

 

[TheShortTimer]

@eyoungren I appreciate the thorough response. Now I understand why I'd always been steered towards CCC or SuperDuper! as a backing up solution on the Mac. I suspect that the recommendations were bourne out of experiences similar to yours...

 

[Certificate of Excellence]

I think making a video a week of life lessons/stories/memories that you want to pass down to your kids would be cool - like a spoken word book of sorts. Making something like that over the decades would be a very cool posthumous gift to your kids/grandkids when we bite the big one. I imagine doing so would take gobs of space to keep and back up as you accumulate videos. I think that would be a great use of your left over space.

 

[klasma]

Porn seems like an obvious choice.

Or rent it out using Storej/Sia/Filecoin/Arweave…

 

[B S Magnet]

So, this isn't a serious issue, just looking for any possible ideas. Between several Macs on my network I have a lot of empty storage sitting around. There are at least four or five Macs I could potentially fill drives up with just stuff.

Things I currently use my storage for:

- Downloads (my best SL Mac Mini is the target for downloads for all my Macs).
- Burn: A 6TB drive on my MacPro is used to store anything I've download and intend to burn (eventually) to disc.
- VMs: Also stored on my MacPro 6TB drive.
- Media: Stored on the NAS, backed up to the G4 PowerMac. The NAS is 6TB and the G4 backup drive is also 6TB. There's still about 3.5TBs left to fill on both drives.
- Tools, long-term storage, OS, frequently used app installs: Also stored on the MP 6TB drive.
- Graphics: Image library, design work, wallpaper images, graphic work. That's on the 2TB RAID and I still have 1.95TB left to fill.
- Scratch/backup: for Photoshop and InDesign/QuarkXPress.

Some other notes. One drive on the MP is 2TB (I have four drives). It's solely being used to store images I take off the internet that I'm interested in but don't rise to the level of graphics I'd use for desktop wallpaper. Drive 4 on the MP is 6TB, 4TB of which is intended for Dropbox. That leaves 2TB for other stuff. Same thing with an external enclosure for the L:09 Mac Mini.

Finally, I have a bunch of 1 to 3TB drives that are just sitting around. I figure those will become backup drives for the stuff I intended to burn, but if anyone has a better use for them I'm listening.

Again, pretty much any suggestion on what to do with my space is valid and welcome. I mean, come on. I'm using one 2TB drive to essentially store junk images!

Not sure if this will be of any help or not, but I can lay out how I have my archive storage managed:

All physical volumes for archival live on my G5, which does storage and file server work on the local network. All physical volumes for archival are configured in RAID 1 arrays (two software, one hardware). There are, at present, three, with plans to break these out to five.

One physical volume is reserved for visual entertainment media (i.e., films, television). This tends to get a lot of read use once items are added to this volume. Files on this device tend to be on the larger side — from the hundreds of megabytes to the tens of gigabytes.

Another physical volume is reserved for software archiving — all the innumerable .dmgs and .zips of stuff I’ve picked up along the way (along with install images of older OS X builds, including all the source file work I have for the SL-PPC project). The sizing of these archives can vary from the very small (in the kilobytes range) to gigabytes. For my own convenience, I set up a top level directory of classes of software.

Another physical volume (for now) was intended to handle long-term, seldom-altered archives — scholarly research and draft work, “reference desk” materials (including documentaries), film photo scanning archives, and deep archives for client work (i.e., past clients/dormant clients get moved to .xz archives).

It is also where my music library and my music video library backups live, as well as where my VM images and system backups live — which is why I prefaced this as “for now”. In the long run, I want music and music video backups to live on a separate, discrete volume more in line with how I manage my television/cinema archives, whilst leaving the reference/client work/academia material where it is. I also want the VM/VirtualPC images to live on their own physical volume — one better equipped to handle constant reading/writing with faster response times. For the VM stuff, I’d love to have it on an SSD RAID 1 volume, but that’s still a long way off as far as budgeting for the sizes I’d need.

Music content generally gets the most activity on this third, sort of onmibus volume (I use VMs less often than most other folks on here, I think), and I’d prefer for these to be in a place which is purpose-prepared for higher activity, but not needing SSD-level speed like with VMs. (The originals for the music and music videos live on another Mac, where I’m actively adding and modifying content, i.e., a lot of writing, then using CCC periodically to back those up to this volume.)

I’m not sure if this is helpful, but this is how I handle things locally.

[Also maybe worth noting that I leave all backing up work to CCC.]

 

[TheShortTimer]

Porn seems like an obvious choice.

I was wondering when someone would suggest this...

 

[Amethyst1]

I was wondering when someone would suggest this...

We are above wasting our time with that kind of content, aren't we?

 

[B S Magnet]

We are above wasting our time with that kind of content, aren't we?

It’s worthy of mention when one is actually producing, directing, and shooting the content (as some friends of mine have been doing for years). It’s kind of banal to mention its consumption when most of the folks around you ::looks around the room:: are basically the key demographic of who consumes it clandestinely anyway (i.e., a lot of y’all consume it, even if you don’t say so, and that’s OK — just so long as you’re paying the creators for it).

 

[eyoungren]

Porn seems like an obvious choice.

Uh…no.

That plus wife will equal divorce.

And after 25 years of marriage with a woman I love very much, two kids (19 and 14) and a mortgage, that isn't worth my time.

Besides…I have my wife.

 

[TheShortTimer]

We are above wasting our time with that kind of content, aren't we?

Um, yeah. Of course...

It’s worthy of mention when one is actually producing, directing, and shooting the content (as some friends of mine have been doing for years). It’s kind of banal to mention its consumption when most of the folks around you ::looks around the room:: are basically the key demographic of who consumes it clandestinely anyway (i.e., a lot of y’all consume it, even if you don’t say so, and that’s OK — just so long as you’re paying the creators for it).

There should be nothing to be ashamed of. I always think of the scene from The People Vs Larry Flynt where Flynt asks why sex is stigmatised whilst war and violence are often celebrated - and yes what's created should be remunerated.

Uh…no.

That plus wife will equal divorce.

Watch it whilst she's out/asleep? (Just kidding)

Besides…I have my wife.

 

[B S Magnet]

Besides…I have my wife.

I’ll be over here on the veranda, lazily imagining @eyoungren saying “my wife” like Borat.

 

[eyoungren]

I’ll be over here on the veranda, lazily imagining @eyoungren saying “my wife” like Borat.

We met at United Parcel Service (same shift) in December 1994. Married in November 1997, moved to Arizona in 2000. First kid came in October 2003 and the second in June 2008.

 

[eyoungren]

Not sure if this will be of any help or not, but I can lay out how I have my archive storage managed:

All physical volumes for archival live on my G5, which does storage and file server work on the local network. All physical volumes for archival are configured in RAID 1 arrays (two software, one hardware). There are, at present, three, with plans to break these out to five.

One physical volume is reserved for visual entertainment media (i.e., films, television). This tends to get a lot of read use once items are added to this volume. Files on this device tend to be on the larger side — from the hundreds of megabytes to the tens of gigabytes.

Another physical volume is reserved for software archiving — all the innumerable .dmgs and .zips of stuff I’ve picked up along the way (along with install images of older OS X builds, including all the source file work I have for the SL-PPC project). The sizing of these archives can vary from the very small (in the kilobytes range) to gigabytes. For my own convenience, I set up a top level directory of classes of software.

Another physical volume (for now) was intended to handle long-term, seldom-altered archives — scholarly research and draft work, “reference desk” materials (including documentaries), film photo scanning archives, and deep archives for client work (i.e., past clients/dormant clients get moved to .xz archives).

It is also where my music library and my music video library backups live, as well as where my VM images and system backups live — which is why I prefaced this as “for now”. In the long run, I want music and music video backups to live on a separate, discrete volume more in line with how I manage my television/cinema archives, whilst leaving the reference/client work/academia material where it is. I also want the VM/VirtualPC images to live on their own physical volume — one better equipped to handle constant reading/writing with faster response times. For the VM stuff, I’d love to have it on an SSD RAID 1 volume, but that’s still a long way off as far as budgeting for the sizes I’d need.

Music content generally gets the most activity on this third, sort of onmibus volume (I use VMs less often than most other folks on here, I think), and I’d prefer for these to be in a place which is purpose-prepared for higher activity, but not needing SSD-level speed like with VMs. (The originals for the music and music videos live on another Mac, where I’m actively adding and modifying content, i.e., a lot of writing, then using CCC periodically to back those up to this volume.)

I’m not sure if this is helpful, but this is how I handle things locally.

[Also maybe worth noting that I leave all backing up work to CCC.]

You're more organized than I am. But my MEDIA library is my largest asset and that occupies just one drive. Your explanation helps though in that it tells me what I can do. Right now the problem is mainly just needing data to fill up drives, rather than the other way around.

A consequence of my haphazard and decentralized approach to storage - although I would say that it's only in the last couple of years that 2 to 3TB drives have proliferated at my house. There was a time where I had one or two 1TB drives and everything else was smaller.

I suppose I should be happy as I've got plenty of space, LOL.

 

[JMStearnsX2]

I have a Synology NAS with 40TB of total storage, only using about have of that. All Macs at backed up via Time Machine to that, also using third party software to clone the Macs drives to a separate share on the NAS. The NAS is then backed up to a 4-disk RAID on my Mac Pro as well as some bare drives I keep offsite.

 

[f54da]

Yeah, as I said having Show Revision History as an option in most Cocoa apps is really great, that relies on Time Machine.

Don't think this is case, you can have show revision history even without time machine ever having been enabled. There are several things going on basically:

* You can have time machine enabled, and it may additionally fetch things from external storage
* You can have time machine enabled, and it may make incremental on-disk backups in the interim when external storage cannot be reached
* Time machine can be disabled, and you can still have revision history for files

The fact that the 3rd option exists surprised even me, I wouldn't have thought to look for it. I think maybe this tops the list of most powerful but most hidden feature. Almost all cocoa-based apps automatically will store versions, even if the app itself may not expose "show revision history" menu. For instance, even though xcode doesn't have this option, underlying source files are still versioned so can open in textedit and see version history through there.

Can see more information about underlying storage format here:

https://arstechnica.com/gadgets/2011/07/mac-os-x-10-7/14/ (it's roughly delta-encoded).

Where are OS X document versions stored? Are they secure?

Support for OS X versioning is spreading among apps (e.g., TextWrangler has just added this feature as of version 4.5), but it is unclear what the security implications of this are. For example, if...

apple.stackexchange.com

Do OS X Lion's Versions and Resume features store the cached data for encrypted .sparsebundle images in a secure way?

I store sensitive information on my laptop in encrypted .sparsebundle images which are made with apps like Disk Utility, Knox and Espionage. The question is, if I upgrade to Lion, when I'm working ...

apple.stackexchange.com

But this means that for things that cannot be delta-encode deasily like images, you can have a chunk of disk space "lost" to these versions.


Apple even offers cocoa api to programmatically access versions of a file. On top of this, others have built useful CLI tool: see https://gitlab.com/jcfields/versions (works even down to 10.7). And also a GUI tool https://gitlab.com/jcfields/restore (uses swift scenes so doesn't work on < 10.10. I tried to rewrite so it would work, but I don't know anything about gui, and I got fed up trying to get cocoa to render a simple table.)

Also closed-source but free VersionsManager on MAS.