Best use of dual Ethernet ports

[michaelbog]

I have a nMP coming soon and would
love to know the best way to use both Ethernet ports. I do not have a switch or a router the supports a aggregated setup. Your thoughts would be great.

Thanks for the time.

 

[evilpaddy]

Link aggregation provides redundancy and not increased speed. At least as i understand it!

Suspect like most people, you will only use the one port!

 

[michaelbog]

That's what I thought, but wanted to check. I have heard of a setup where one went to the local network and one went out to the internet.

But cannot find more info other than that.


Have a good one.

 

[maflynn]

What kind of internal network are you running? Do they all go through a single router? Splitting the traffic in that way probably doesn't make sense if you have a single router that handles all your networking needs.

 

[Cubemmal]

I have four, the two internal and two on the dual TB monitors.

I use link aggregation on the internal ones. The other two aren't doing anything at the moment, but I'm thinking of allowing access to some of my restricted VLAN's. But that's your choice, either aggregate, VLAN or different subnets.

 

[jasonvp]

Link aggregation provides redundancy and not increased speed. At least as i understand it!

There are 2 ways to do link agg, and 1 does indeed increase speed. You turn 2 GigE ports into 1 2xGigE port, effectively. In order to do that, you need a managed switch that can handle 802.3ad bundling.

 

[ibgb]

There are 2 ways to do link agg, and 1 does indeed increase speed. You turn 2 GigE ports into 1 2xGigE port, effectively. In order to do that, you need a managed switch that can handle 802.3ad bundling.

No, this is not factual. This has been discussed multiple times on this forum.

Do you think both ethernet ports are simultaneously used for a single file transmission? No. Each tcp transmission occurs using a single ethernet pair (send/receive); it doesn't split them up. If you get lucky, you can have 2 separate tcp transmissions using 2 ethernet pairs simultaneously--but this is not the same a a 2gig pipe. A file sent say by ftp will only get about 120MB/s maximum speed. You won't be getting 240MB/s for a file send.

http://www.cisco.com/c/en/us/td/docs/ios/12_2sb/feature/guide/sbcelacp.html#wp1053763

 

[shaunp]

Or don't bother with link aggregation and just have two separate ports on separate IP networks. Use one for all your daily 'stuff' and the other for TM backups.

 

[antonis]

There are 2 ways to do link agg, and 1 does indeed increase speed. You turn 2 GigE ports into 1 2xGigE port, effectively. In order to do that, you need a managed switch that can handle 802.3ad bundling.

Hm...shouldn't the "other side" to also be able to send/receive at that speed, though, in order to achieve that ?

 

[jasonvp]

No, this is not factual. This has been discussed multiple times on this forum.

I would re-read my first post in this thread very carefully before trying to correct me. At no point did I say that a single TCP stream could be hashed over multiple GigE links. I merely stated that an 802.3ad link bundle WILL provide the Mac with 2xGigE's worth of bandwidth. There's nothing in that statement that is incorrect. It's up to the OS and upstream switch to hash the respective outbound traffic in order to take proper advantage of both links. This is obviously much easier to do when it's a many-to-one or one-to-many style conversation.

For what it's worth: you don't need to quote Cisco's garbage to me as I've been a (rather successful) network engineer for the greater part of the last 2 decades. I actually know what I'm talking about regarding this topic, versus a lot of forum participants who only think they do.

 

[paulrbeers]

There are 2 ways to do link agg, and 1 does indeed increase speed. You turn 2 GigE ports into 1 2xGigE port, effectively. In order to do that, you need a managed switch that can handle 802.3ad bundling.

http://www.ieee802.org/3/hssg/public/apr07/frazier_01_0407.pdf

See Page 7....

----------

I would re-read my first post in this thread very carefully before trying to correct me. At no point did I say that a single TCP stream could be hashed over multiple GigE links. I merely stated that an 802.3ad link bundle WILL provide the Mac with 2xGigE's worth of bandwidth. There's nothing in that statement that is incorrect. It's up to the OS and upstream switch to hash the respective outbound traffic in order to take proper advantage of both links. This is obviously much easier to do when it's a many-to-one or one-to-many style conversation.

For what it's worth: you don't need to quote Cisco's garbage to me as I've been a (rather successful) network engineer for the greater part of the last 2 decades. I actually know what I'm talking about regarding this topic, versus a lot of forum participants who only think they do.

Then you need to be more careful with your posts, because I read it the exact same way. There are many here that would take what you said and assume they get a 2gb port based on your explanation....

Understanding a concept and explaining it so that others can understand (fully) are two entirely different skills....

EDIT: Actually you know what, your original explanation is so wrong and you tried to cover it up. You stated you get 1x2gb port. No. You always end up with 2 x 1GB ports. That's it. The difference is that your computer can carry two different "conversations" at one time.

It's like your computer is playing chess with two different people at one time (which by the way, I am no good at doing). Both games continue at the same pace, but you accomplish both games at the same time.

 

[snberk103]

I have the previous generation MP with two ethernet ports. One connects to the network/internet - the second ethernet port connects to my printer. Using the network connection on the printer allows more flexibility in its location since the ethernet cable can be longer than the USB cable.

 

[jasonvp]

EDIT: Actually you know what, your original explanation is so wrong and you tried to cover it up. You stated you get 1x2gb port. No. You always end up with 2 x 1GB ports. That's it. The difference is that your computer can carry two different "conversations" at one time. .

Paul, you're a bit outside of your league here. The OS sees it as a single link.

mrfreeze$ ifconfig bond0
bond0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=b<RXCSUM,TXCSUM,VLAN_HWTAGGING>
	ether 3c:07:54:7c:6e:1a
	inet6 fe80::3e07:54ff:fe7c:6e1a%bond0 prefixlen 64 scopeid 0x8
	inet 169.254.200.77 netmask 0xffff0000 broadcast 169.254.255.255
	nd6 options=1<PERFORMNUD>
	media: autoselect (1000baseT <full-duplex>)
	status: active
	bond interfaces: en1 en0

It's a single link, with a single MAC address. It's up to the OS and switch to do the hashing, but it still appears to everything in and on the OS as a single link (in my case, I'm also running VLANs which is why there aren't any useful IPs on it).

 

[paulrbeers]

Paul, you're a bit outside of your league here. The OS sees it as a single link.

mrfreeze$ ifconfig bond0
bond0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=b<RXCSUM,TXCSUM,VLAN_HWTAGGING>
	ether 3c:07:54:7c:6e:1a
	inet6 fe80::3e07:54ff:fe7c:6e1a%bond0 prefixlen 64 scopeid 0x8
	inet 169.254.200.77 netmask 0xffff0000 broadcast 169.254.255.255
	nd6 options=1<PERFORMNUD>
	media: autoselect (1000baseT <full-duplex>)
	status: active
	bond interfaces: en1 en0

It's a single link, with a single MAC address. It's up to the OS and switch to do the hashing, but it still appears to everything in and on the OS as a single link (in my case, I'm also running VLANs which is why there aren't any useful IPs on it).

I disagree. It might show as 1 link, but it can not use the entire bandwidth which is what you implied (or maybe more appropriately didn't explain well enough). It requires two "conversations" in order to max the bandwidth. In essence it is still 2 x 1GB "tunnels".... Therefore my analogy is spot on. I person doing two things at once. No one process will complete any faster than if it was done singularly, but rather they both can be done at "full speed' at the same time.

Who gives a @!#$ how the OS sees it, if the end result is that you can't transfer a single file any faster than if it wasn't tied together. For your average end user this is what they care about and where they get messed up and your explanation did nothing to explain that.

 

[jasonvp]

Who gives a @!#$ how the OS sees it

Actually, that's precisely what matters: how the OS sees it. That you don't understand that is basically the crux of our disagreement and it requires that you have some networking skills, knowledge, and real world experience. You have none, as is evident by this thread.

While I may not have properly explained my original post, I wasn't incorrect. Try as you might, you won't be able to prove otherwise.

 

[chirpie]

It IS St. Patricks day. Maybe share a beer instead of the building angst. Cheers!

 

[beaker7]

Actually, that's precisely what matters: how the OS sees it. That you don't understand that is basically the crux of our disagreement and it requires that you have some networking skills, knowledge, and real world experience. You have none, as is evident by this thread.

While I may not have properly explained my original post, I wasn't incorrect. Try as you might, you won't be able to prove otherwise.

I like how you guys are arguing pedantic minutia about a point that is completely irrelevant to the thread. The dude doesn't have a switch that supports it as stated in the OP.

Gotta love MacRumors.

 

[michaelbog]

Or don't bother with link aggregation and just have two separate ports on separate IP networks. Use one for all your daily 'stuff' and the other for TM backups.

This is what I was hearing and I am wanting to do. Is there are discretion on how to set this up.

Mb

 

[shaunp]

This is what I was hearing and I am wanting to do. Is there are discretion on how to set this up.

Mb

Just use separate IP networks, that would be enough to keep the backup traffic off the rest of your network.

 

[michaelbog]

Ok, thanks

 

[snarfquest]

Does the MacPro / Mavericks support LACP? I've been trying to figure that out but so far all my search results just find lots of people very confused on the difference between interface bonding and a LACP channel. They interchange the 2 as if they were the same thing.

 

[michaelbog]

Does the MacPro / Mavericks support LACP? I've been trying to figure that out but so far all my search results just find lots of people very confused on the difference between interface bonding and a LACP channel. They interchange the 2 as if they were the same thing.

Here is what I have found.

http://support.apple.com/kb/PH14045?viewlocale=en_US&locale=en_US


You need a switch that supports Link Aggregation Control Protocol (LACP)


Still looking for a good how to on setting up a multiple IP network.


I hope this helps,

Mb

 

[bxs]

It's interesting that the OP asked this question...

Over past weekend my goal for bonding all 3 MacPros (a 4-core Nehalem, an 8-core Nehalem and a new 12-core MacPro 6,1) at my son's office was successful. My only difficulty was figuring out the IP address of the Cisco switch so I could monitor the link states and obtain statistics of packets being moved over each port and for any errors. There were no errors on any of the links. I disconnected one of each of the bonded ethernet ports to make sure failover worked and that presented no issues.

The switch I used was the Cisco Small Business SG200-18 1 Gigabit switch. It's noiseless (has no fan) and runs quite cool. After 3 hours of use it was slightly warm on ts top surface. This switch has the capacity to process 36 Gbps (just a tad over 4 GBytes/sec) and will support IEEE 802.ad LACP up to 4 groups of dynamic link aggregation, and has a remote web browser-based management providing configuration, system dashboard, system maintenance and monitoring... Very nice indeed. The switch was purchased from NewEgg for $207. I check with Cisco and NewEgg is a certified reseller for Cisco switches. Here's a ref to the switch... http://www.newegg.com/Product/Product.aspx?Item=N82E16833150120. Interestingly enough, I noted that as soon as I placed my NewEgg order for this switch they increased it by $1.00 - very weird.

The Cisco Small Business 200 Series Smart Switch Administration Guide states on page 116 the following

"Link Aggregation Control Protocol (LACP) is part of IEEE specification (802.3az) that enables you to bundle several physical ports together to form a single logical channel (LAG). LAGs multiply the bandwidth, increase port flexibility, and provide link redundancy between two devices.

Two type of LAGs are supported: Static and Dynamic"

I decided to configure Dynamic LAGs for each of the 3 MacPros in my Son's office.

A LAG supports Load Balancing: Traffic forwarded to a LAG is load-balanced across the active member ports, thus achieving an effective bandwidth close to the aggregate bandwidth of all the active member ports of the LAG. Traffic load balancing over the active member ports of a LAG is managed by a hash-based distribution function that distributes Unicast and Multicast traffic based on Layer 2 or Layer 3 packet header information.

This Cisco managed smart switch has 18-ports. I configured things as follows using the switch's management web-based software which was easy to use. I actually configured the switch at home several days before going to my Son's office.

I used CAT6 wires everywhere, although Cisco indicated CAT5 would be sufficient so long as they were not 100s of feet long.

This Cisco switch can have up to 4 LAGs configured. I only wanted to use 3 for the 3 MacPros that each had 2x 1GbE ports.

With this many cables it's important to label the cable ends, especially the ends that plug into the switch.

Port 1: A Router line from a 1TB TC's LAN port.
Port 2: Connected to a 27" iMac named Saturn
Ports 9 & 10: connected to MP6,1 named Titan (these were bonded to aggregate the line data speeds and for failover).
Ports 12 & 13: connected to 8-core MP named Mercury (these were bonded to aggregate the line data speeds and for failover).
Ports 15 & 16: connected to 4-core MP named BAZ (these were bonded to aggregate the line data speeds and for failover).

I tested as follows

1) Created a large file named Test_file on Titan's internal 1TB PCIe-based flash storage
2) Setup File sharing on Titan so that Test_file could be accessed by other Macs
3) On each Mac (Saturn, Titan, Mercury and BAZ) I performed a 'time cksum Test_file' at same time

Using Activity Monitor on each Mac I observed the Network's Data received/sec and Data sent/sec

On Titan the Data sent was a steady 380 MBytes/sec
On Mercury the Data received was a steady 145 Mbytes/sec
On BAZ the Data received was a steady 155 Mbytes/sec
On Saturn the Data received was a steady 60 Mbytes/sec

4) I now ran on Mercury, BAZ and Saturn in turn a 'time cksum Test_file' command

On Mercury the Data received was a steady 150 Mbytes/sec with periodic bursts up to 280 MBytes/sec
On BAZ the Data received (did not test this Mac)
On Saturn the Data received was a steady 85 Mbytes/sec with periodic bursts up to 124 MBytes/sec

I was quite happy with these results.

My son wanted me to make the iMac read from Titan at same rate as the 8 and 4 core MPs could. I told him to replace the iMac with a MP. The poor olde iMac has but one ethernet port...

Now all we need is for Apple to develop TCP/IP over Thunderbolt.... (ha ha).

The bonding (Virtual Interface) done using the Network Prefs was very straight forward. I did note that these Virtual interfaces took close to 30 to 60 secs to form a connection and suspect the delay was associated with the Cisco switch figuring things out.

The one problem I was not able to figure out was being blocked from having the Cisco switch system display its CPU utilization.

The Cisco switch and the 4 Mac configuration has now been running solidly for 4 days without issues. Over the next few weeks I will be gather data from the Cisco switch's internally kept statistics to see if errors are occurring and to understand how much data traffic is being sent/received. I will also want to see the size of data packets being routed around to know whether I should configure for jumbo frames or not to improve data send/receive rates.

 

[CH12671]

We had an HP server that had dual NIC's. We didn't have a switch that did link aggregation (or, more precisely, I didn't have the login for the switch to configure it). So I assigned each NIC a separate IP address, and had half my clients login (through script) to shares through one IP address, and half through the other. This, in effect, split up the demand between the two NIC's, allowing simultaneous requests without crowding (assuming the two requests were coming in on different NIC's). Had I been able to aggregate it, the switch and OS would have hashed that out for me, and I could have pointed everyone to a single IP address....
The way I did it, it was still possible to have two requests come in on the same NIC, therefore not giving me any real advantage at all. However, if two clients each were mapped to different NIC's on the server, then the server could carry on those two simultaneous conversations without crowding the bandwidth on either NIC.

Multiple IP network can be as simple or complex as you want it to be. What two things do you want it to do? I assume one will be internet? What else? If it's simply a printer, or NAS unit, you can just assign it an IP scheme (i.e. 10.1.0.xxx) and put the printer/NAS on the same IP scheme, and they will see each other, but be separate from your other network.

Otherwise, two NIC's are good for putting one machine on two different subnets if that is something you or your office needs. Lots of possibilities, but most people will probably just use one.

 

[RCZ]

If you run WIndows 2012 R2 Server or Windows 8 as the client OS, and the backend is also Windows 2012 R2 server, you can benefit from the dual links, as the scale out file services will use as many links as possible for transfers.. this is called SMB Multichannel – Permitting multiple SMB sessions to be established simultaneously over single and/or multiple high-speed network connections to maximize the utilization of these network storage paths. SMB Multichannel automatically detects when multiple sessions can be managed over 1GbE and faster network connections and enables this feature to fully utilize these paths.

not sure if the built-in adapters also support RDMA.. probably not..