Organizing (for back up) a MASSIVE amount of stuff

[Commode]

I have about 30TB (largely very redundant, though, and I'm guessing <10TB total of content) of old drives, some intact, some recovered and just a mess of files, and cumulatively this mess contains my work, school work, hobbies, multiple music libraries, photography, videos, etc. from the past 20 years.

It's all a mess, I've been terribly disorganized and now there's backup after recursive backup so huge amounts of redundancy and a lot of files in older formats (Claris Works, etc.) that I will have to save in another format on an old iMac I bought that runs classic.

Multiple users, too.

How the heck do I get started?

Eventually I want to reduce this backup into files I find interesting or worthwhile, which will be a very tiny percentage of them.

I bought a 16TB RAID array to start with. I was thinking of working drive-by-drive then:

Documents
Movies
Photos

Then by year maybe?

I am so overwhelmed.... and disorganized.... any organized people out there?

I wish I didn't have to organize each drive individually but I think cumulatively it's too much.

Any way to track progress effectively? I guess I could label everything after I've archived it.

 

[ssmed]

Duplicate Detective Cleaner works quite well to reduce some of the bulk, but these projects are huge effort and you need to decide what is necessary and be harsh and use the trash when you can. I would use Lightroom or Aperture for the photos and for such items use a really comprehensive naming system e.g. 2019-05-20 George Dog 54201.NEF keeping the original name at the end to help with duplicate checking.

Also ensure you are using RAID for protection not speed (e.g. RAID 5 or 6 etc)

 

[Commode]

The drives are so redundantly backed up already that I doubt I will lose any data. I suspect I need the full 16TB just to fit everything, but I won't be storing on the RAID array thereafter. Will move to separate drives (redundant, non-RAID) once condensed down.

I was planning to organize each drive separately and then from there merge into one big (but comprehensive) organized backup of everything, eliminating redundancies along the way, and from there parse down to only things I actually value. But this might be a little silly. I have a LOT of old iTunes libraries... But now I have Spotify Premium lol. So what's the point. But I don't have any of those artists and playlists saved so I want to add them back to my library on Spotify as I go. About 90% of the point of this is the journey–I already have the data safe, it's just such a mess I don't know what it even is and I'm curious to read my old school papers, print some old photos, etc. I'm uncovering files I'd thought I'd lost and realizing there's little point to a backup if I can't even figure out what's in it.

Unfortunately my cameras (Sigma DP2 and DP3 Merrill) aren't compatible with Lightroom.

Once I narrow things down I'll be very harsh with trashing stuff. I already have the rest of the data, I only want to remove the chaff from it, but I need to identify chaff first. But in the future I plan to delete a lot more than I save.

Do you have any ideas that are more elegant than marking things in a certain color after they've been transferred to a drive in order to keep track of progress?

Is it smart to create a prospective final structure? I think ideally I should structure each organized backup the same so when I merge them the folders correlate cleanly.

I have been very disorganized and am working on it. This is a tough thing for me. Borderline learning disability of some sort. :/

 

[InuNacho]

I split up my stuff by type on different drives and back it up on one big local drive and smaller drives at relatives homes.
For example, in my 6 bay Mac Pro I have a drive solely for photography, a drive for video production, a drive for media which will soon be split up for just iTunes and another for other stuff, a drive or docs, and a drive for misc stuff. It's all backed up on a large 10tb drive that sits on my desk.

I then once every few months clone the drives to small 1-2tb externals and drop them off in my parent's garage.

I don't delete anything. I pay to stream music on my phone, but I don't bother do that on my computer since I have music I can't get anywhere else on there.

 

[zqbobs]

I have been dealing with a similar situation as yours, although on a smaller scale. Keeping track of progress is a key element because you can't do it in one session, as you're aware. I've used a large HDD (e.g. Seagate Expansion 8TB) and copied all my smaller disk content to that before finding duplicates, organizing by type, etc. I find having everything in one place (drive) makes it easier. Back it up (or at least keep your originals until you're done), as others have warned!

A couple of app helpers:

NeoFinder (https://www.cdfinder.de/) - a great disk cataloging program, for online and offline disks. Can do thumbnails for images, find duplicates, etc. Much faster than Finder for searching. Well supported. I've used it for over 10 years.

Phoenix Slides (http://blyt.net/phxslides/) - very fast image bulk viewer, including JPEGs,RAWs, etc. Can rotate, delete images.

 

[Commode]

Thanks, everyone.

I'm going to look into NeoFinder and Duplicate Detective Cleaner.

Copying everything to another drive and then organizing those files before evening moving on to the next drive is a very smart idea. I was trying to do multiple drives sequentially and began to fear I was skipping files without realizing it.

I have a few drives that were recovered and have no folder directory at all and that are composed, in one case at least, primarily of Classic files (which I need to move to another computer and open in ResEdit to determine the creator, which often I can't do because there's no database I know of that lists ResEdit creators), so I don't even know how to deal with that. Some really weird OS 8/OS 9 system stuff that I should probably just ignore lest I become too obsessive and give up before I even start...

Thanks again for the advice. I'll research and I suspect will buy at least one of the apps recommended here.

 

[Tech198]

I have about 30TB (largely very redundant, though, and I'm guessing <10TB total of content) of old drives, some intact, some recovered and just a mess of files, and cumulatively this mess contains my work, school work, hobbies, multiple music libraries, photography, videos, etc. from the past 20 years.

It's all a mess, I've been terribly disorganized and now there's backup after recursive backup so huge amounts of redundancy and a lot of files in older formats (Claris Works, etc.) that I will have to save in another format on an old iMac I bought that runs classic.

Multiple users, too.

How the heck do I get started?

Eventually I want to reduce this backup into files I find interesting or worthwhile, which will be a very tiny percentage of them.

I bought a 16TB RAID array to start with. I was thinking of working drive-by-drive then:

Documents
Movies
Photos

Then by year maybe?

I am so overwhelmed.... and disorganized.... any organized people out there?

I wish I didn't have to organize each drive individually but I think cumulatively it's too much.

Any way to track progress effectively? I guess I could label everything after I've archived it.

Half is always, usually same photo, music, exact version of songs etc..

You can bring it down by 1 tenth, or perhaps more..

After that get a NAS and/or biggest external portable drive (4TB) you can..

(free) : I've always found dupeGuru good: https://dupeguru.voltaicideas.net/

 

[Commode]

Half is always, usually same photo, music, exact version of songs etc..

You can bring it down by 1 tenth, or perhaps more..

After that get a NAS and/or biggest external portable drive (4TB) you can..

(free) : I've always found dupeGuru good: https://dupeguru.voltaicideas.net/

Thanks, I've been using dupe guru and it's been really helpful (and free is always great).

One thing I notice is that it doesn't work with files I've recovered from a corrupted drive, perhaps the recovery process is responsible for this.

Does anyone know of any software that visually matches photos? I have downscaled versions, etc. too that I likely no longer need and it would be nice to match them visually. It is perhaps a terabyte of photos. :/

 

[MareLuce]

Very helpful topic. Thanks to all who have replied with
- their approach
- what tools have worked for them