Need advice on a new setup for my external hard drive

[Martinpa]

So, I've been having a lot of issues with my external hard drives for a while now, and I was thinking of re-working how it all works and wanted to pick other people's brain on how good of a setup it'll be for my needs.

I currently have 4 ext. hard drives:

• 4TB USB 3: Music and Video Libraries, and Comics
  • I've been migrating to Apple Music, so when all the missing songs have been transferred, I'll probably make a backup of my library on a smaller driver, just in case, and clean up some space on that drive.
• 4TB USB3: All my raw photos for Lightroom, and Photoshop files
• 5TB USB3: Various documents
• 5TB USB3: Time Machine Backup

I have found a Mac mini 2014 (so 4 USB ports on the back) for 180$CAN and I'm thinking of getting that and placing it in the living room, plugged into the TV and directly into my modem/router (everything else in my apartment is connected via wifi). I'd then plug all my ext. hard drive into that Mac mini and access their content from my main iMac through Bonjour (I think that's right...) I hardly think it would be an issue for my Music, Comic and Video libraries, what I'm more unsure about, is if I'd be able to work on my raw photos on Lightroom from the hard drive through my internet connection. Would Time Machine also be ok in that setup? I'm also a video editor, but I'll probably keep working off of work drives plugged directly into my main computer, though some assets might be hosted on one of those drives (sound FX library, some video overlays, etc.)

I'm encountering increasing amount of little issues here and there on my iMac and I'm seriously thinking of doing a clean install of the OS and start fresh (currently doing a backup), so I'm thinking now might be the time to really just put a lot of energy in getting this right and organize my externals correctly (as it stands, I can't really have more than one plugged in at once, so it makes that near impossible.

Any thoughts or advice?

 

[rm5]

Servers. There's your solution. I know what you're thinking—really? A SERVER? But read this entire post so you know exactly why I think you need a server(s).

There are two issues with your proposed setup:

1. The 2014 Mac mini is deathly slow
2. Just the thought of running multiple external drives off of one (or even two for that matter) USB busses—that's gonna be SUPER SLOW

So no, this is not the solution. What IS the solution IMO is to do what I did and get some servers. You don't have to get anything fancy—just some generic HP or Dell servers that you can stick in another room (they are a tad noisy). You can just get a server without the drives, and add them yourself—you can have as much storage as you want, because servers are scalable. Keep in mind though of two things:

1. The noise - the more servers you add, the more noise. Also the more drives you add, the more noise.
2. EVERY COMPUTER (including the server) MUST be plugged into Ethernet!! This will NOT work if they aren't!

So with those two things in mind, go on eBay and grab a cheap (but decent) server, or two, or however many you need. Don't spend more than $200 on one server, unless absolutely necessary (I think my PowerEdge cost $250 plus the 10 hard drives which is a little extreme for such old hardware TBH).

Also keep in mind that in my experience, data access speeds are around the same as a normal 7200 RPM hard drive in a RAID 5 setup (which is the best IMO because 2 drives can fail and you'll still be up)—so around 200 MB/s—not great, but doable. I've edited 4K off of my servers with no issues. And you wanna know the best part? Since my servers are overkill in terms of CPU (both are 12-core systems with high-capacity, high-speed disks), I can be copying files on one machine while editing 4K on another machine, while AT THE SAME TIME, backing up via a TM "shared folder"—with no slowdown!

I know this is probably not the answer you're looking for, but what do you think? Is this feasible? I think it'll be SO MUCH BETTER for you in the long run. Now of course, you could go the Mac mini route, but just don't. Just don't. You'll be happier otherwise I think.

EDIT: One problem with servers though is that the RAM acts as cache space—so when the cache runs out, it slows to a CRAWL... so it would be advantageous for you to get some SSDs to use as cache!

EDIT #2 - then you just create different TrueNAS "shares" for different purposes. Have your Lightroom/Ps stuff on one share, files on another share, etc. You could even create different "virtual drive sets" on the server's RAID controller and separate them that way, anything goes.

 

[Martinpa]

Servers. There's your solution. I know what you're thinking—really? A SERVER? But read this entire post so you know exactly why I think you need a server(s).

There are two issues with your proposed setup:

1. The 2014 Mac mini is deathly slow
2. Just the thought of running multiple external drives off of one (or even two for that matter) USB busses—that's gonna be SUPER SLOW

So no, this is not the solution. What IS the solution IMO is to do what I did and get some servers. You don't have to get anything fancy—just some generic HP or Dell servers that you can stick in another room (they are a tad noisy). You can just get a server without the drives, and add them yourself—you can have as much storage as you want, because servers are scalable. Keep in mind though of two things:

1. The noise - the more servers you add, the more noise. Also the more drives you add, the more noise.
2. EVERY COMPUTER (including the server) MUST be plugged into Ethernet!! This will NOT work if they aren't!

So with those two things in mind, go on eBay and grab a cheap (but decent) server, or two, or however many you need. Don't spend more than $200 on one server, unless absolutely necessary (I think my PowerEdge cost $250 plus the 10 hard drives which is a little extreme for such old hardware TBH).

Also keep in mind that in my experience, data access speeds are around the same as a normal 7200 RPM hard drive in a RAID 5 setup (which is the best IMO because 2 drives can fail and you'll still be up)—so around 200 MB/s—not great, but doable. I've edited 4K off of my servers with no issues. And you wanna know the best part? Since my servers are overkill in terms of CPU (both are 12-core systems with high-capacity, high-speed disks), I can be copying files on one machine while editing 4K on another machine, while AT THE SAME TIME, backing up via a TM "shared folder"—with no slowdown!

I know this is probably not the answer you're looking for, but what do you think? Is this feasible? I think it'll be SO MUCH BETTER for you in the long run. Now of course, you could go the Mac mini route, but just don't. Just don't. You'll be happier otherwise I think.

EDIT: One problem with servers though is that the RAM acts as cache space—so when the cache runs out, it slows to a CRAWL... so it would be advantageous for you to get some SSDs to use as cache!

EDIT #2 - then you just create different TrueNAS "shares" for different purposes. Have your Lightroom/Ps stuff on one share, files on another share, etc. You could even create different "virtual drive sets" on the server's RAID controller and separate them that way, anything goes.

Sounds more… involved than what I had in mind, especially since it’d be very hard to wire everything around the apartment for Ethernet.

 

[rm5]

Sounds more… involved than what I had in mind, especially since it’d be very hard to wire everything around the apartment for Ethernet.

It's actually much easier than it might seem to configure a server—I had mine up in running in less than an hour. But I understand the difficulty of having to wire everything... but that would be the case even if you used a Mac mini.

Just for some idea of how slow access is gonna be on wifi - you'll be lucky to get more than 10 MB/s read and write speeds over WiFi. That ain't gonna change if you decide to do with the Mac mini OR the server - you just need an Ethernet connection.

 

[pmiles]

NAS is what you need. I have a simple 2 drive NAS myself. Plugged into the router. Two 4TB drives functioning as separate drives, although they could have been striped together in a single RAID.

Drawback to this is your current hardware is likely not NAS rated and would need to be formatted for the NAS even if they were rated prior to use. In other words, you'd need to get both a NAS and the number of drives you need to go this route.

A NAS is quite simple to setup. I can access my video collection and audio collection from my computer or my Apple TV or my Sonos. One box houses it all.

Why are all of your files split across so many different drives? I can see photos requiring a large storage capacity, but documents? Seems a little overkill for some of it (5TBs ?). Sounds like you are using physical drives like partitions.

As for working over the network... really not the best idea. Store your files locally while being edited, then copy them to the NAS when done. You best throughput is always local... not to mention horsepower. You can work over the network, it's just not the prudent thing to do.

As for watching movies or listening to music, a NAS is perfect for these tasks.

You could jerry-rig a bunch of separate drives to function in a similar fashion as a NAS, but it won't be pretty. A NAS is just a simple an elegant solution to what you need. You could just continue as you have been connecting and disconnecting external drives as needed, it's essentially the same thing only less convenient/elegant.

As with any storage solution, you have no less or greater risk of lost data with a NAS. You will still need to employ a backup strategy that meets your personal needs.

 

[badgerbadgerx2]

Consolidate and simplify. My advice/recommendation/whatever:

Get a 2 bay RAID case -- combine the two 5 TBs into a single 10 TB RAID 0. Put everything on it. Depending on drives, (assuming they are basic 5400 RPM drives) might get close to 300 MB/s, more than enough speed. Dont spend a lot for case, under $100?

Pickup the largest single drive you can afford for a TM backup.

Or any combo. Combine 4 TB into a single fast 8 TB RAID 0, use one of the 5 TBs as backup.

Done.

You could also go with a second RAID 0 case and the 2 4TB drives, for a 8 TB backup... but Im probably a little less comfortable recommending 2 RAID 0s. 2 RAID 0 cases might be $180.

 

[Fishrrman]

"Any thoughts or advice?"

Yup.
Do not ... repeat, DO NOT buy a 2014 Mini, particularly one with only 4gb of RAM.
Get something newer.

I'd suggest either an Apple-refurbished 2018 Mini, or perhaps an Apple-refurbished 2023 m2 Mini (with 16gb of RAM and a 512gb SSD) when they hit the online store in another few months.

The 2018 Mini is a very trusty Mac.
Again, 16gb of RAM should be "your minimum".

 

[rm5]

The disadvantages to an "off-the-shelf" NAS:

1. Not scalable
2. Doesn't support nearly as much RAM as a server
3. WAY more expensive than a server
4. Limited OS capability and/or availability
5. Bandwidth not nearly as much because you're working off far fewer drives

As much as people disagree with me, a server is the solution.

 

[pmiles]

The disadvantages to an "off-the-shelf" NAS:

1. Not scalable
2. Doesn't support nearly as much RAM as a server
3. WAY more expensive than a server
4. Limited OS capability and/or availability
5. Bandwidth not nearly as much because you're working off far fewer drives

As much as people disagree with me, a server is the solution.

1. It is scalable... a 4 bay NAS with only a single drive in it can have 3 additional drives added WITHOUT affecting the original drive. You don't have to use a RAID setup with a NAS. I added the second drive to my NAS down the road without any impact to the original drive. You can even upgrade the RAM without impacting the setup.
2. You don't need a ton of RAM since it functions primarily as storage. A simple multimedia rated NAS can handle 4K video et al on the fly. Again, heavy lifting should be done locally on the base computer, not over the network.
3. My NAS setup (2 bay) with just the one drive cost under $500 US. Relatively cheap all things considered.
4. I can connect a Mac, a PC, or a UNIX box to my NAS on the fly without issue.
5. Typically anything run over a server is slower than running it locally. In all my years working with computers and massive data files, I never ran any of my applications on the server, only locally. The server was ALWAYS slower.

There are use cases for a server and use cases for a NAS. Most home use is better suited to a NAS. Now if you have a base computer with a woefully inadequate amount of storage to begin with, you're going to want to connect a drive to it to make up for that shortfall. You could install your application on the external drive and run it there... but again, it would be slower than had it been run locally on the base system. Having a bunch of drives to store data is not the same thing as needing a bunch of drives to actually manipulate the data. Most people just need a place to store data when not being used, not a place to work with large amounts of data 24/7.

Not saying to go with a NAS or to go with a server... just saying there are options out there... of which the current solution the OP is using is just as viable in this equation. Not pretty, but it works.