Video file management questions about hard drives for videos

[Reminisce32]

Going to be a first time mac user with a 2016 MBP with a 2 TB SSD and would like some direction on the best practices for having my video files backed up in a couple places while also making video editing as efficient as possible. I need to be able to manage about 150GB of new video/photo files each month and I'd like it all backed up long term.

My questions are as follows:

Is it true that video editing is faster if you have one drive where the editing software runs, second separate drive for temp files/cache, and a third drive for where video files are kept?

What harddrive products (compatible with 2016 MPB 15") can help me most time-efficiently copy my video files to 2+ separate harddrives in terms of transfer speed as well as perhaps somewhat automating the whole process either on a preset schedule or at least making it so I only have to copy the files in one action? And are there any TB3 options? If not, when can they be expected?

As a PC user with usually limited space, I never really had the ability to back my actual PC up consistently. What's the best way to do this with a mac? An Airport device?

Any other advice is appreciated.

Thanks!

 

[catonfire]

Reminisce32,

Multiple drives are the way to go. Avoid reading from and writing to the same drive at the same time.

1. Boot Drive: Laptop SSD.
I only use this for applications and autosave folders. Admittedly I use my laptop in clamshell mode at a desk 90% of the time. If you're editing on the road, you may need to use the Laptop SSD for several purposes but it's not maximal.

My understanding is that it's a bad idea to fill up spinning HDDs past a certain % as they like some headroom. SSDs might be similar in this regard so don't fill it up if you don't have to. 2TB can be overkill if you just store apps on the Boot Drive. I have about 150 apps and 512 GB has been plenty sufficient for me. I usually have about 200GB free.

However, if you will be on the road and think you will need that storage then it might be a good idea. Otherwise, you're really paying a premium for that onboard storage which would never be the best place to keep data in a multi-drive editing system which is optimal if you are working in a fixed location.


2. Cache Drive: SSD
Use a separate SSD, or OWC / Akitio / etc. RAID for your media cache which you can set up in the Preference panel of your various apps. I think the Akitio Thunderbolt 3 enclosures are out. Be careful with compatibility. Some T Bolt 3 peripherals on the market right now aren't compatible with this latest MacBook implementation of T3. There was an article on MacRumors. I think it was hubs not enclosures but consult with manufacturers before purchasing. 512GB to 1TB might be sufficient for something like Resolve, Premiere, After Effects, Photoshop, and Lightroom simultaneously. You can always clear your cache monthly.

Sandisk, Samsung are reliable.


3. Media Drive: HARDWARE HDD RAID / SSD RAID / SSD
If you only have 150 GB a month per project, you are fortunate in that you can get away with SSDs or SSD RAIDs for your media drives and go for blazing speed over storage size.

On the other hand if you will need access to old media on a rolling basis, have 4K work, multicam work, or just want to be prepared for impromptu jobs, then invest in a large Thunderbolt 3 RAID. Go for as many disks as you can afford. You're looking at >$2k-$3k for 12, 18, or 24 TB RAIDs. You'll want Hardware RAIDs not software RAIDs. Use RAID 5 mode for both speed and data protection.

Usual Suspects: Promise, Areca, LaCie, etc.


4. Project Drive: SSD /HDD
One reason to keep projects on yet another drive is redundancy. Don't keep autosave files on the same drive as your primary project files because if that drive tanks, you lose everything.

Secondly, you can also render outputs to this drive, as you wouldn't want to write to your Media 'read' drives if you don't have to.

Thirdly, you will eventually amass a lot of project files which will then be enough to slow down your boot drive if you stored it all there.

Fourthly, your data is more secure if you don't keep it on your boot drive. I don't know what kind of work you do but if you ever had to take your laptop in for repair, all your files could end up in the hands of strangers and you might not have it backed up. I also find it a little more efficient to clone stand alone drives over having stuff mixed in on a boot drive. It makes reconciling a clean 1 to 1 operation.


5. Backups: Bare Hard Drives, Toasters
If you can get a Thunderbolt 3 'toaster' drive dock from OWC, that would be the fastest most efficient solution. The current T2 dock is about $279. 150 GB a month isn't that much though and a T2 dock should chew through that in short order. Might as well wait for the new T3 toaster if it is forthcoming and similarly priced though.

Backup Hard Drives pile up, so I don't bother with enclosures. I keep nearline bare hard drives in plastic cases in a fireproof data safe for onsite storage. I keep offline bare hard drives at an offsite location which I swap out once a month with my on site backups. Remember though, when handling bare drives, always touch a metal surface to get rid of any static from your body first and never ever touch the SATA connectors.

Slower HDDs are OK for backups since they just sit in a safe 90% of the time. But since this is your archive, get the ones with the highest reliability rating on NewEgg or wherever.

I back up video media at the time of ingest and if critical, bring in my offsite backups at this time as well.
I think it's a good idea to always have 3 copies of your media on 3 separate drives at all times. All the more so when your primary storage is a RAID.


Chronosync
I use Chronosync software for backups. You can create 'custom backup templates' or you can leave a backup drive attached and it will backup on schedule. For small daily backup of project files, I actually just save to a thumb drive plugged in. I only pull archive drives out of their safe once a week or monthly.

The great thing about Chronosync is that it has many different methods for copying data since people have different ways of managing their files. And it will partition off and save folders that don't jibe so you can doublecheck them later.

It's a little confusing at first but once you test practice a few procedures the advantages and utility of each backup style will make sense.


Pathfinder
This is 3rd party file system browser overlay on top of Mac OS which I find greatly enhances productivity, file management, and backing up. It allows a great deal of customization of the interface. Of late Apple has been stripping out a lot of great features it used to have, like color labeling for folders and files. Pathfinder puts all these things back and more. I find the double-pane browser essential to reconciling backup drives with primary drives.



6. Workstation Organization
Where do all these drives go? I store all my drives on IKEA hutches or attached shelves underneath my workdesks. The more SSDs you can employ the better things will be heatwise. Sometimes I put a mini fan on the RAID When it gets hot. Same for the laptop. But it's cooler under the desk and whatever noise they generate is away from your ears. A battery backup and good surge suppressors for all these peripherals can be useful to.

Check out Adobe or Creative Cow forums for more precise hardware knowledge.

 

[joema2]

....2016 MBP with a 2 TB SSD...need to be able to manage about 150GB of new video/photo files each month and I'd like it all backed up long term....Is it true that video editing is faster if you have one drive where the editing software runs, second separate drive for temp files/cache, and a third drive for where video files are kept?

That is the standard advice but it does not apply to hyper-fast PCIe SSD like the 2016 MBP. You can put everything on the internal drive and get good editing performance (from and I/O standpoint) -- until you run out of space.

2 TB is not that big for serious video editing. Besides the media files themselves, editing produces scratch files, temp files, render files, proxy files, etc -- which can take several times the original media size. You will probably be OK for a little while but you'll need to plan for external storage.

...What harddrive products (compatible with 2016 MPB 15") can help me most time-efficiently copy my video files to 2+ separate harddrives in terms of transfer speed as well as perhaps somewhat automating the whole process either on a preset schedule or at least making it so I only have to copy the files in one action?

MacOS has a built-in full-featured backup program -- Time Machine. All you need is a hard drive and it will continuously back up the entire system, while allowing selective browsing and rollback of older files/folders.

Many people rely only on Time Machine for backup. However for serious work IMO you ideally need a separate off-line backup besides Time Machine. I use Carbon Copy and a separate set of backup drives: https://bombich.com/

..And are there any TB3 options? If not, when can they be expected?...

You will now face the issues of the 2016 MBP USB-C-only design. There are several USB-C portable hard drives available but they (obviously) are not natively backward-compatible with all other computers having only USB-A: http://www.imore.com/best-usb-c-hard-drives-mac

For older USB devices you will need adapters. However we already face compatibility issues from the multiple USB-A/B standards. I have to take a bag full of various USB connectors on site. Eventually USB-C will eliminate all those so a single connector/cable type will interconnect and charge most devices. This is a good video summarizing it: https://www.cnet.com/how-to/usb-type-c-thunderbolt-3-one-cable-to-connect-them-all/

...As a PC user with usually limited space, I never really had the ability to back my actual PC up consistently. What's the best way to do this with a mac? An Airport device?...

Unlike Windows PCs which don't come with decent backup software, all Macs come with Time Machine which is one of the best backup programs ever invented. You will get fastest performance backing up to a USB 3 or Thunderbolt drive. I have several of these Seagate Backup Plus Fast drives which are internally RAID-0: https://amzn.com/B00HXAV0X6

 

[ColdCase]

Your 2TBSSD is external? What is the capacity of your internal PCI SSD memory? Over what period of time do you need to routinely edit a set of source video. What editing software will you be using?

You can use time machine to backup your internal drive to one or more external drives. Simple and effective. Use something like Carbon Copy Cloner to backup your external media drives to one or more other drives. Thats more efficient for more static files you need to keep backed up.

What many of us do, using FCPX for example, is set up a working library on the fast internal PCI SSD drive or, if the internal is too small, an external "working" Thunderbolt SSD (look for faster SSDs like Samsung 850s). All the app and cache data is on the internal. Import your working video there (or the "working" SSD). When done, or when it reaches a convenient size, move that library off to another reliable external media drive for archive purposes (remember to back up the media archive). Then start over again with a new working library (its not uncommon to have several working libraries for different projects/subjects). I usually share (export) the final video for distribution to a drive different than the one where the library is stored.

When I'm using something like handbrake or compressor to transcode, its more efficient to have the source on one SSD, and the destination on another SSD.

Specifics depend on your workflow, is that 150GB HD video and compressed such that it may need to be expanded to pro res for efficient editing, for example?

There is a lot of good old school advice around that works well with old school constraints and rotational HDs and RAID, but for a small video project like yours, they have been mostly OBE.

150B of source data is small, I usually grab 200GB in gopro videos a week plus go through another 200GB just as a hobby. You may need a naming convention to keep things straight, especially if you want to go back and find something six months down the road.

 

[RCAFBrat]

Going to be a first time mac user with a 2016 MBP with a 2 TB SSD and would like some direction on the best practices for having my video files backed up in a couple places while also making video editing as efficient as possible. I need to be able to manage about 150GB of new video/photo files each month and I'd like it all backed up long term.

My questions are as follows:

Is it true that video editing is faster if you have one drive where the editing software runs, second separate drive for temp files/cache, and a third drive for where video files are kept?

...

Keeping program files, media and temporary files separate applies when dealing with HDD rather SSD.

(disclaimer: while I understand the principals described below I haven't bothered to double check the terminology)

While the speed of the drives is certainly a big consideration, the thing that is most overlooked is latency ... think about how a HDD works and this becomes really easy to understand ... the HDD incorporates heads that read data on the spinning platters and these heads must be physically repositioned to read different blocks of data including the file system index each time it has to read or write a different file. This repositioning can be minimized or done in parallel when different HDD are used for different purposes.

An SSD does not have any moving parts so latency is virtually eliminated compared to HDD.

This combined with the fact that a RAID array of HDD is no longer required to achieve fast read / write performance means that using that internal 2 TB SSD will give you far superior performance as long as you are systematic in terms of your workflow with regard to backups and archiving project files to external HDD so you have capacity for current projects.

My two cents!

 

[kohlson]

When SSDs were first introduced they used the same SATA-III interface as most HDDs - 6Gbps. SSDs were faster in the sense that they had essentially zero latency. Since then, new PCIe-based interfaces have widened the gap. The SSD module in the latest MBP is reported to have 3GBps speeds. That is, no latency and about 5x the throughput. It will be interesting to see the lab tests on the new systems.

 

[MacUser2525]

Lots of good information here I would add a second to the recommendations of Pathfinder and CCC wonderful programs that I have used forever in addition to that for file management DefaultFolderX from Saint Clair Software. It extends the save dialog allowing you to have a favourites folder short cut (that contains your pre-defined set of folders) among other things to save to from the save dialog. And as has been mentioned as well backups, backups and more backups you can never have enough copies of your important data. Personally I have gone with three a main machine that contains master copy of everything then two additional machines backed up by a rsync script I fire up when needed to sync them up, all use the zfs file system on a raidz1 configuration of the drives in them.

http://www.stclairsoft.com/DefaultFolderX/