Need help to set up a server solution

[yoak]

Hi

I'm not sure what I need, so please bear with me and hopefully I can get some advice here.

I run a small production company and I'm a DP (cameraman).
I shoot a lot HD footage, 1080, 4K and 5K so I'm starting to fill up a lot of hard drives.
I need somewhere to store all my footage, and ideally I a should be able to have access to it from everywhere so I can download clips where I am if needed.
It obviously can't be public to anyone without my permission.
It has to be fast enough so I can edit from it when I'm at the office, so a NAS won't do.
My editing station is a Mac Pro, but I would like to be able to hook up my MacBookpro to it when needed.

Limitations:
Cost is an issue to some extent and my macs dont have thunderbolt yet.

What solution do you suggest?
One thought crossed my mind, that I could have a NAS as a secon unit serving both as back up and the unit I would access my files from remotely in addition to a faster storage for editing, but ideally I would like to be able to access the fast unit remotely.
Will a mini server do what I need, or cold the macpro serve as a server as well as an editing machine? (guess it would use a lot of electricity if it has to be always on)

 

[miles01110]

It has to be fast enough so I can edit from it when I'm at the office, so a NAS won't do.

Assuming you're connecting from the office in the first place you're still going to have to pass all the data through the network connection, suggesting that an NAS would in fact work just fine for you.

 

[yoak]

Assuming you're connecting from the office in the first place you're still going to have to pass all the data through the network connection, suggesting that an NAS would in fact work just fine for you.

I've tried editing from NASs in the past and they don't work well as a scratch disk for HD editing. They are too slow, even the ones that claim to work for editing. I even had 2tb caviar black WD disks inside, but after 15 minutes the system choked

 

[DustinT]

The problem is that anything that connects to your Mac's over Ethernet will be limited to the performance of the Ethernet. So, a good NAS unit will work just fine. You can expect about 80-100 megs a second, which isn't bad at all. That's fast than USB 2.0 by about 3x and it should be able 1.5x faster than firewire 800. Some NAS's support NIC teaming which allows you to use 2 network ports to transfer your data, so it's basically twice as fast as I've posted above.

The only thing that will be faster is Thunderbolt, e-sata, USB 3 or fiber channel. Thunderbolt isn't an option without buying new Macs, E-sata won't work with your MacBook Pro, neither will USB 3.0 or Fiber Channel (well, you can but it's really, really expensive.)

So, your options are to get a USB 3.0 device and a USB 3.0 card for your Mac Pro. The performance should be just fine. The problem would be when you want to edit on your Macbook you'll be limited to USB 2.0 speeds which won't be fine. In fact, I don't think you could work with that, realistically.

E-sata will work on your Mac Pro but not your Macbook Pro. Well, there are adapters to make them work but they rely on USB 2.0 so there's no advantage there at all.

The exception to what I've posted above is this: If your MacBook Pro has the ExpressCard slot you can grab an ExpressCard to USB 3.0 or E-sata adapter and use the device at full speed. If you can do it, that's a great option.

Gigabit Ethernet is your best bet for cross platform compatibility. If you use NIC teaming, you can see some properly fast numbers and it's a very workable solution.

 

[mainstay]

Why not just load your mac pro with some nice, high capacity drives, withhigh access rates (enterprise drives) and access them locally? No network, no transfer.

When you need to access them remotely, you just vpn into your MP and use that terminal or a session in that workstation.

I would still have some sort of NAS for backup purposes, but this would not be your primary data holder. You may be able to re purpose your old drives for the NAS?

This is going to be your cheapest answer with the highest immediate performance. And it should allow you to have a few other stations in the office also using the same files. Beyond say 6 other stations, you will need to start budgeting for something more serious.

 

[yoak]

Why not just load your mac pro with some nice, high capacity drives, withhigh access rates (enterprise drives) and access them locally? No network, no transfer.

When you need to access them remotely, you just vpn into your MP and use that terminal or a session in that workstation.

I would still have some sort of NAS for backup purposes, but this would not be your primary data holder. You may be able to re purpose your old drives for the NAS?

This is going to be your cheapest answer with the highest immediate performance. And it should allow you to have a few other stations in the office also using the same files. Beyond say 6 other stations, you will need to start budgeting for something more serious.

My Mac Pro has been filled up twice already with 1-2tb drives, and I take one out and replace it with a new one. Then comes the problem of having to swap that old drive back in again when I need some footage on it.
I think ideally I would have the drives inside the Mac raided as scratch disk,and then a NAS as a backup connected to the Mac that I can connect to wirelessly if I need access to the footage from somewhere else.
BUT, if I do this, don't I have to have my Mac pro on all the time when I'm away?

----------

The problem is that anything that connects to your Mac's over Ethernet will be limited to the performance of the Ethernet. So, a good NAS unit will work just fine. You can expect about 80-100 megs a second, which isn't bad at all. That's fast than USB 2.0 by about 3x and it should be able 1.5x faster than firewire 800. Some NAS's support NIC teaming which allows you to use 2 network ports to transfer your data, so it's basically twice as fast as I've posted above.

The only thing that will be faster is Thunderbolt, e-sata, USB 3 or fiber channel. Thunderbolt isn't an option without buying new Macs, E-sata won't work with your MacBook Pro, neither will USB 3.0 or Fiber Channel (well, you can but it's really, really expensive.)

So, your options are to get a USB 3.0 device and a USB 3.0 card for your Mac Pro. The performance should be just fine. The problem would be when you want to edit on your Macbook you'll be limited to USB 2.0 speeds which won't be fine. In fact, I don't think you could work with that, realistically.

E-sata will work on your Mac Pro but not your Macbook Pro. Well, there are adapters to make them work but they rely on USB 2.0 so there's no advantage there at all.

The exception to what I've posted above is this: If your MacBook Pro has the ExpressCard slot you can grab an ExpressCard to USB 3.0 or E-sata adapter and use the device at full speed. If you can do it, that's a great option.

Gigabit Ethernet is your best bet for cross platform compatibility. If you use NIC teaming, you can see some properly fast numbers and it's a very workable solution.

As said before, I (and others) have tried editing HD footage using NAS over Ethernet and they are not fast enough.
A NAS is not suited for editing, even if they are fast enough in theory.

Esata could be the way to go as I do have the ExpressCard slot and I could just get a Esata card in my macpro.

Is there a way to wake my Mac pro remotely so I don't have to have it on all the time?

 

[mainstay]

Whenever we've encountered this issue we just add external drives via firewire, usb 3, or esata, (and now I suppose, thunderbolt, although I haven't actually used it myself).

Virtually unlimited expansion capabilities for MINOR cost.

The other option would be to use old systems as storage servers.

You can also use a NAS as a long term storage device and pull your active projects to the MP as you need them, then archive them when done.

There are high speed NAS devices that utilize 10/100/1000 connections effectively. QNAP comes to mind... this allows for easier deployment in a network with multiple terminals needing access to the material.

Honestly, the cost of a drive is so minimal that you could put each project on a set of drives and physically store them in your safe. When you need access to that material later on you can readily drop them into an external bay or hot drive.


>>BUT, if I do this, don't I have to have my Mac pro on all the time when I'm away?


How much is electricity where you live?

The monitors consume almost as much juice as a MP, so leave those off when you are away and you will drop your consumption in ~half.

Apple reports a 2008 MP 2.8 8-core as consuming 155W on idle and 318W at Max CPU.

At $0.10 / KW-hr that is $0.37 a day to have remote access to your system...

If your office can't accept $12 a month in server costs then you could cut that back further by having set sleep and wake times.

My systems turn off at midnight (unless I am working on them in which case I cancel the sleep timer when it pops up) and then restart at 6:30 am and run a series of diagnostics, virus checks (for my Windows systems), and backups. When I arrive at 7:00/7:30 am, things are ready to rock and roll.

You could also look at waking the computer on demand via magic packets, but I usually just leave them running. It's part of the cost of doing business.

 

[yoak]

Whenever we've encountered this issue we just add external drives via firewire, usb 3, or esata, (and now I suppose, thunderbolt, although I haven't actually used it myself).

Virtually unlimited expansion capabilities for MINOR cost.

The other option would be to use old systems as storage servers.

You can also use a NAS as a long term storage device and pull your active projects to the MP as you need them, then archive them when done.

There are high speed NAS devices that utilize 10/100/1000 connections effectively. QNAP comes to mind... this allows for easier deployment in a network with multiple terminals needing access to the material.

Honestly, the cost of a drive is so minimal that you could put each project on a set of drives and physically store them in your safe. When you need access to that material later on you can readily drop them into an external bay or hot drive.


>>BUT, if I do this, don't I have to have my Mac pro on all the time when I'm away?


How much is electricity where you live?

The monitors consume almost as much juice as a MP, so leave those off when you are away and you will drop your consumption in ~half.

Apple reports a 2008 MP 2.8 8-core as consuming 155W on idle and 318W at Max CPU.

At $0.10 / KW-hr that is $0.37 a day to have remote access to your system...

If your office can't accept $12 a month in server costs then you could cut that back further by having set sleep and wake times.

My systems turn off at midnight (unless I am working on them in which case I cancel the sleep timer when it pops up) and then restart at 6:30 am and run a series of diagnostics, virus checks (for my Windows systems), and backups. When I arrive at 7:00/7:30 am, things are ready to rock and roll.

You could also look at waking the computer on demand via magic packets, but I usually just leave them running. It's part of the cost of doing business.

First of all, thanks to everyone posting. It helps getting things into perspective.

We tried the Qnap TS-859 Pro+, but it didn´t work for HD editing.
Adding external storage as I need is what I´m doing now, but it turns in to a bit of a mess as I often end up with more than one project on each drive.
Just looking at my desk now I have a fully loaded MP and 6 external desktop drives and 4 portable ones and I´m tiered of the mess and the time it akes me to find the right drive sometime when looking for older footage.

I guess I could use part of your advice and store each project on backup disks in a safe or another location (as I kind of do now) and then just hook up a NAS as an archive and server where I have access to everything, but not use it as a scratch disk.

It wasn´t so much the cost of electricity as the wear of the MP I was thinking of, but maybe thats a non issue?

Last, but not least; How do I (in an easy way) access my MP and the (future) NAS from my MBP when on the road?

 

[mainstay]

VPN server + Screen Sharing

Or LogMeIn.

Or VNC clients....

 

[DustinT]

Yes, I think you're getting the right idea now. You need a file server or NAS, just don't try to use one as your scratch disk.

Honestly, you may want to give some serious thought to a pro-level solution. The extra drive bays really pay off in the long run. Have 8-12 bays will give you a small mountain of storage potential. Plus, the speeds will be as good as possible.

 

[Foogoofish]

Yes, I think you're getting the right idea now. You need a file server or NAS, just don't try to use one as your scratch disk.

Honestly, you may want to give some serious thought to a pro-level solution. The extra drive bays really pay off in the long run. Have 8-12 bays will give you a small mountain of storage potential. Plus, the speeds will be as good as possible.

Check out the Drobo lines....best in show at the moment for enterprise imo.

 

[Waragainstsleep]

It sounds like you should be looking at a proper RAID array. If you can find one that supports fibre channel and iSCSI, you should be able to use it with both Macs easily enough until you can upgrade the laptop to get Thunderbolt and add fibre to that too.

If you can't stretch to a new one, scour eBay for a used Promise VTRAK unit and supply your own disks if you need. There was one a while back that went for a stupidly low price with SAS drives in it.

 

[jasonvp]

I guess I could use part of your advice and store each project on backup disks in a safe or another location (as I kind of do now) and then just hook up a NAS as an archive and server where I have access to everything, but not use it as a scratch disk.

My Mac Pro is set up as follows:

• SSD as for OS and applications. The SSD is in the 2nd 5.25" bay, leaving the 4 other SATA slots open.
• 2x300GB 10K SATA drives configured as a RAID0 volume, purely for scratch space and the OS's paging space (if it needs it)
• 2x1TB drives in a RAID1 array for local storage of stuff that I'm working on
• 3TB RAID10 NAS attached via my GigE LAN

While editing, I keep everything local to the Mac. When I'm done, I save everything off to the NAS for safe keeping.

Last, but not least; How do I (in an easy way) access my MP and the (future) NAS from my MBP when on the road?

My solution may not work for you; it depends on how much money you want to spend and what kind of Internet connectivity you have local to you. I have a business class cable connection at home (50 down/5 up) with fixed IP addresses. I run my own router on it, and can get to/from the Mac and the NAS with ease that way, via ssh. Using any other protocol to connect remotely is risky and foolish IMHO.

jas

 

[yoak]

Hi Jas
I don't have More than 12 down, but 5 up.
Would those speeds enough you think?

Your MP set up is similar to how I want to set up mine in the future
How big is your ssd?
I have the 5770, how is the 4000 working for you?
Is it difficult to set up and administer your router solution?

Thanks

 

[jasonvp]

I don't have More than 12 down, but 5 up.
Would those speeds enough you think?

Probably, as long as you're the only one using it. You could run very low-use web servers with that kind of bandwidth, but moving around tons of files and whatnot would start to tap into that bandwidth. Would you be the only one using it?

How big is your ssd?

It's the OWC 120G unit. Far bigger than it needs to be; I could have made do with the 60G one. Literally the only things on it are the OS and apps. I've moved the /Users directory off to the 1TB RAID1 volume. The swap space (normally /var/vm/swapfile) is sitting on the 600G RAID0 volume. However, the rig has 48G of RAM, and so far it's never needed to swap.

I have the 5770, how is the 4000 working for you?

Perfectly well. I bought it because I use Premiere Pro for editing, and there are tasks I do with that software that can be punted to the CUDA cores on the card. It helps quite a bit w/speed.

Is it difficult to set up and administer your router solution?

It wasn't difficult for me. It's a small mini ITX-based machine running Linux on a thumb drive. The only thing it's doing is acting as a router and packet filter. I don't know what you're UNIX skill set is, so I don't know if it would be easy for you to set up.

jas

 

[yoak]

I've had to start using premiere as well when I got my RED. It Is such a time saver. I've heard the 4000 should help quite a bit, but I've also heard people having problems with it.

Which NAS do you use?

BTW I have no experienc with unix, so that's out of the question

thanks again

 

[jasonvp]

I've had to start using premiere as well when I got my RED. It Is such a time saver. I've heard the 4000 should help quite a bit, but I've also heard people having problems with it.

There are a lot of rendering that Premiere can send to the CUDA cores. You'd have to verify with Adobe (via their online resources) whether what you're doing can be done with the CUDAs. I'm merely taking 1080p AVCHD video from two cameras and setting up a picture-in-picture with them. Then scaling the entire thing down and saving it as an MPEG4 file. The scaling bits are all handled by the video card. The encoding has to go to the CPUs.

Which NAS do you use?

Home-built running Linux. I rarely buy a solution when I know I can build a better one for less money.

jas