What's the best practice LAN setup for video editing on a high speed transfer and workflow?

[mabok]

Hi,
Any senpai able to help me on setting up 2 - 6 mac desktop for high speed networking? This is for video editing workflow.

Here's my purpose of setting up:
To transfer huge file sizes on a shorter amount of time.
- File sizes is around 250gb or more. Most of it contains video clips and such.

When using software e.g: Adobe suites, Final Cut Pro, etc...
I would like to use some files from another pc while doing the work. Without experience lagginess and such.

My current LAN setup is basically each mac is connected using ethernet port on one another, like a ring loop style.
pc 1 > pc 2 > pc 3 > pc 4

What if pc 1 want to 'tap' in pc 4?

Those are my requirements. Would appreciate the help guys! Thanks.

 

[techwarrior]

You actually have a few things to decide to get this right.

Network:

It is inefficient to link Mac to Mac directly. For #1 to Access #2, the cable must be connected between the two, then to connect to #3, the cable must be physically connected to #3.

If you use a network switch and\or router, all Macs can access all of the other Macs without having to change cables.

A router will also be able to manage IP Addresses on each of the Macs, so they are unique. If Internet access is required, a router also gives all of the Macs access to the internet. Many routers have a USB port, enabling a shared drive to be attached and accessible to all users on the network.

If you go the router + USB route, I would be sure to get a USB3.0 Drive, and a router with USB3.0 port(s). USB 2 speeds are about 450 Mbps where USB 3 is about 5Gbps. Of course, the drive itself will make a difference, an SSD will be much faster than a spinning HD, but capacity\cost are considerations. Make sure the router has 1Gbps ports for Ethernet.

Shared Location:

You can share from each Mac, but then a user account\password would need to be setup on each of the Macs and the folder(s) where the shared content are stored would need to be available for read and write for these accounts. Sharing from the router can be done with a single shared login, or even without security. If the Macs share, then you have to make sure users save files in the right place when done, if central storage is used, they are always stored in the same place.

Finally, a NAS is an option as well. A NAS is a network storage device. Many have RAID (redundant disks) enabling you to have instant backups of all of the data. depending on speed, level of resiliency, a NAS could become costly. But, if the loss of data is a concern, NAS can be a great solution. With a 2 disk NAS, data on disk 1 can be mirrored to disk2, so if 1 drive fails, the other still has a good copy. With a 3+ disk NAS, you can get into more advanced RAID schemes which not only increase resiliency, but also speed up read\write speeds. Some NAS can even use an SSD drive to cache read\write so you get faster drive speeds, the SSD then writes the data to the hard drives when activity permits.

It all kind of depends on how much you are willing to spend to tackle this. A Switch can run around $20-30, a router maybe $75-200. A USB Drive, $100-200. A NAS, $200-1000+. You may need a router + a switch, most routers have 4 LAN ports, and a NAS would need 1 (or 2) LAN ports. A NAS with 2 LAN ports can be setup to increase bandwidth\speed to and from the NAS, but would require a router than supported bonded NICs. Essentially, a bonded NIC is two network interfaces sharing the same IP Address, and many home routers don't support bonded NICs. but, what this would do is speed up transfer speeds if say two users were opening or saving two different video files and the same time.

So, a bit more information on what you already have (routers\switches), and how much you are willing to spend and you will get more specific guidance.

 

[Mikael H]

A NAS is a network storage device. Many have RAID (redundant disks) enabling you to have instant backups of all of the data. (...) But, if the loss of data is a concern, NAS can be a great solution. With a 2 disk NAS, data on disk 1 can be mirrored to disk2, so if 1 drive fails, the other still has a good copy.

@mabok: An important thing to add to the above:
A (mirroring or parity-based) RAID will increase your resilience against specifically hard drive failures. It isn't by itself a backup, and any investment in storage (RAID capable or not) should also include a budget for enough separate storage to fit actual backups of the data stored.
To be very clear: Any file operation with unwanted consequences will be immediately replicated across the drives in a RAID set, so unless you also have actual backup copies of your data, you will get bitten sooner or later.

 

[mabok]

I don't see using a RAID is a need for my workflow.
We have external hdd for backup files (raw footages, online video, and such)

I'm wondering for @techwarrior , multiple of mac connected to a router...
how's the speed will be like when transferring lets say 250gb worth of video clips from pc #1 to pc #2?
I want to move on from previous workflow where we usually transfer files via external hdd (e.g: pc #1 copy files to external hdd, then external hdd connected to pc #2 and copy&paste)

Also any idea will the network be congested if,
Pc #1 open a final cut pro and the footages is at pc #2?

This is by using your suggestion, the normal multiple pcs connected to a router/switch on a 1gbps.

 

[Mikael H]

Over a 1Gbps network link, your file transfers will max out at ~100 MB/s, which is the same ballpark as a regular external HDD, but quite a bit slower than an SSD over a modern interface.

If you purchase the cheapest consumer gear, the maximum throughput of the network switch backplane will be a bottleneck if you run network traffic concurrently between multiple computers. On the bright side, already at a price point starting around $100 per 8 ports, the switch backplane shouldn't be a bottleneck at all.

The next potential bottleneck is hard drive speed. If you have mechanical drives in your computers, they themselves max out at only a slightly higher speed than what can be transferred across the network, so anyone working hard on the computer while another computer uses resources from their hard drive will notice a slowdown.

 

[techwarrior]

as Mikael H suggests, network speeds will be impacted by the quality of network gear you choose. In addition to the 1Gbps Ethernet ports, each switch (or router) has a backplane that carries data from port to port. A 12 or 24 port switch generally has more backplane capacity than a 4 or 8 port switch.

Where this comes into consideration is when 2 or more PC are actively opening files across the network. If ports 1 and 2 are transferring a file at up to 1Gbps over the network, and another PC on port 3 and 4 is doing the same, up to 2Gbps are being passed through the switch backplane. For your needs, most consumer grade 8 port switches will have adequate capacity, for instance this one claims to have 16Gbps switching (backplane) capacity:

https://www.amazon.com/NETGEAR-Ethe...4471668&sr=8-5&keywords=gigabit+8+port+switch

When you connect PC <> PC, the speed is 1Gbps. When you connect PC <> Switch <> PC, the same 1Gbps is available for the transfer provided all activity on the switch doesn't exceed the backplane capacity.

The slowdown is generally the disk read\write speeds. A typical HDD transfers read\write at around 100-120MBps or just under 1Gbps. An SSD transfers read\write at around 500MBps or 4Gbps. The fastest load times for files would be local SSD storage where the SATA3 bus carries data from SSD to CPU at about 6Gbps (real speed would be the 4Gbps SSD access speed). Loading a local file from HDD would tend to be in the range of 1Gbps or 4x the load time of SSD. When you start putting the data on a network with 1Gbps speeds, the transfer rate will slow down to the 1Gbps speed, so HDD vs SSD makes less difference. These are max theoretical speeds, reality is often a bit less, so SSD storage will tend to be slightly faster than HDD, but at what cost?

The main reason for considering central storage is ease of access and backup. Lets say you add a WD MyCloud 4TB single Drive NAS (about $250). There would be no resiliency, but all files would be stored in a single location. A second 4TB USB drive could be connected to this to do automatic backups, another $100 or so. The drive in this case is HDD and the network interface is 1Gbps. So, if your users typically open one file at a time, the speed of this solution would be similar to the speed of accessing files on any of the Macs, but would slow down a bit when 2 users are opening or saving files at the same time (due to limits on the network and drive access). Same would be true if 2 users were accessing files on the same PC at the same time.

NAS can speed things up in 2 ways. First, with multiple drives, the read\write speeds can be improved because your read or write a portion of the data from each drive instead of all the data from one drive. Say you have a 4 disk RAID5 array. You write data to 3 drives simultaneously, or effectively, you use 1/3 of the drive i\o capacity for a given read\write operation. The 4th disk contains a parity bit that is used to rebuild if one of the drives fails. RAID 5 is a useful method to both speed up read\write speeds, and to provide resiliency if 1 disk fails, but if 2 fail at the same time, your out of luck unless you have a backup strategy. So, if 2 users are reading\writing at the same time, there is more drive i\o capacity available.

The second way a NAS can speed things up is network speeds. Many NAS are capable of bonding\teaming\aggregating 2 1Gbps NICs. This results in 2Gbps network capacity to and from the NAS. So, if 2 users are accessing the NAS at the same time, each has access to a full 1Gbps of transfer speeds. To do this, the router or switch needs to be able to handle bonded NICs, this is generally not available on consumer grade gear.

Some NAS have 10Gbps network interfaces, and with a switch that is capable of 10Gbps ports, the network throughput to and from the NAS can be increases 10X. Most Macs cannot use 10Gb NICs, but this would allow each Mac to have a full 1Gbps access to the NAS if all 4 are reading or writing from the NAS. Because the disk access speeds would be slower than the network, an SSD cache on the NAS would help speed up file access speeds.

The point is, there are ways to speed things up, add resiliency, and simplify management of the files. It depends on how much you are willing to spend to attain these goals.

The cheapest route is simply networking the 4 Macs with 1Gbps switch\router. You still have to maintain discipline to do backups, and users need to know where files exist, and have credentials to access the files on the various Macs. Central storage puts all the assets in one place for easier access, management, backup, but might be slightly slower than accessing directly on a Mac if two or more users are opening or saving files at the same time. Adding NAS with RAID can speed things up a bit, and adding NAS with higher capacity network speeds and SSD cache can greatly improve file access speeds.

So, again, it comes down to what you are willing to spend to achieve your goals.