Ethernet Link Aggregation

[1nfr4r3d]

Hi Guys,

has anyone managed to successfully set up an aggregated ethernet link on a Mac Pro?

I am running a server which i'm trying to set this up on and have a Dell PowerConnect 2716 which i can manage and login to via browser and set up a link aggregated group by combining ports 1 + 2. So far so good.

I then go through the network preference pane on the Mac Pro and set up a virtual interface combining the 2 ethernet ports into 1 bond / LAG. This works for all of about 90 secs then drops the connection saying the port on the switch doesn't have 802.3ad link aggregation enabled???

Does anyone have any suggestions what i can try as i am fast running out of ideas

 

[funkahdafi]

Hi Guys,

has anyone managed to successfully set up an aggregated ethernet link on a Mac Pro?

I am running a server which i'm trying to set this up on and have a Dell PowerConnect 2716 which i can manage and login to via browser and set up a link aggregated group by combining ports 1 + 2. So far so good.

I then go through the network preference pane on the Mac Pro and set up a virtual interface combining the 2 ethernet ports into 1 bond / LAG. This works for all of about 90 secs then drops the connection saying the port on the switch doesn't have 802.3ad link aggregation enabled???

Does anyone have any suggestions what i can try as i am fast running out of ideas

It works fine here. I am using HP Procurve and Cisco switches, works with both. I figure your Dell config might not be right.

Does the connection drop while you have traffic on both links? If not, maybe you need to set keepalive or something like that (if the Dell allows you to do so).

 

[1nfr4r3d]

hi funkahdafi, thanks for getting back to me.

The interface doesn't give any other options other than specifying which ports you wish to aggregate into a group.

The thing is, if i disconnect one of the cables from the switch and reconnect it, i get the LAG working albeit briefly. After around 90 seconds or so the Mac just loses it and says link aggregation is not supported on the switch?

Is it possible the switch itself is faulty or are these switches just not up to the job? Do you know of any way in which i can test these things?

 

[funkahdafi]

hi funkahdafi, thanks for getting back to me.

The interface doesn't give any other options other than specifying which ports you wish to aggregate into a group.

The thing is, if i disconnect one of the cables from the switch and reconnect it, i get the LAG working albeit briefly. After around 90 seconds or so the Mac just loses it and says link aggregation is not supported on the switch?

Is it possible the switch itself is faulty or are these switches just not up to the job? Do you know of any way in which i can test these things?

It's hard to tell without being able to take a look at the switch. I don't think it's faulty (else the aggregate channel wouldn't come up at all I guess). I don't know Dell switches so I can't even give you any hints on where and how to do some troubleshooting.

To me it sounds more like a compatibility issue between your Mac and the Dell. Make sure your Dell properly supports the LACP protocol (the 802.3ad part of the error message you see). If Dell use some other, proprietary method, than you won't be able to bond the two together. If the Dell actually does support LACP, then we need to look into logs or whatever the Dell offers for troubleshooting.

 

[AlteMac]

For those of us who are less informed, can someone please explain what aggregating ethernet ports means and why one would want to do it? Thanx.

 

[funkahdafi]

For those of us who are less informed, can someone please explain what aggregating ethernet ports means and why one would want to do it? Thanx.

Link Aggregation let's you bond together multiple ethernet interfaces. The bond is called an aggregate and offers loadbalancing of the data (so with 2 gigabit ports you actually get 2 gigabit speed). It also offers fault tolerance so that one of the links fails, traffic will still flow over the other links. The aggregated interfaces appear as one virtual/logical interface to the OS.

It's usually used in datacenters for speed/bandwidth and redundancy reasons. At home, only true freaks use it

Here is some more detailed, high level information on the topic:
http://en.wikipedia.org/wiki/Link_aggregation

 

[1nfr4r3d]

hi funkahdafi,

as far as i know it does support 803.2ad properly - however i will find some Dell documentation to confirm this.

I will pick this up over the weekend as the server was out for a whole day yesterday and the company pretty much stood still all day - i wasn't the most popular person in the building.

Thanks for your help so far, i'll do some more digging over the weekend and let you know what i find.

 

[funkahdafi]

hi funkahdafi,

as far as i know it does support 803.2ad properly - however i will find some Dell documentation to confirm this.

I will pick this up over the weekend as the server was out for a whole day yesterday and the company pretty much stood still all day - i wasn't the most popular person in the building.

Thanks for your help so far, i'll do some more digging over the weekend and let you know what i find.

Haha.... I can't even count how many times I was the most unpopular person. **** happens is what I say.

Good luck!

 

[nanofrog]

It's hard to tell without being able to take a look at the switch. I don't think it's faulty (else the aggregate channel wouldn't come up at all I guess). I don't know Dell switches so I can't even give you any hints on where and how to do some troubleshooting.

To me it sounds more like a compatibility issue between your Mac and the Dell. Make sure your Dell properly supports the LACP protocol (the 802.3ad part of the error message you see). If Dell use some other, proprietary method, than you won't be able to bond the two together. If the Dell actually does support LACP, then we need to look into logs or whatever the Dell offers for troubleshooting.

This is the sense I'm getting as well.

I took a look at Dell's Technical Specifications for the 2716, and it shows this:

IEEE 802.3, IEEE 802.3u, IEEE 802.1D, IEEE 802.1Q, IEEE 802.3ab, IEEE 802.1p, IEEE 802.3x, IEEE 802.3ad (LACP)

(Source).
Not seeing anything about a proprietary implementation anywhere (Tech Specs only), but not seeing anything on 802.1AX either. So digging into the User Manual would be warranted (MP and any other NIC port's specs too).

I'm not sure what other equipment may be involved either, as there could be a mis-match situation somewhere (not all links are duplex, or there's a static aggregation issue for example <802.1AX not supported on all ends>).

Past looking into the above, I'm not sure (screen shots of the settings might help to start, after reading up on that unit via the User Manual).

as far as i know it does support 803.2ad properly - however i will find some Dell documentation to confirm this.

Thanks for your help so far, i'll do some more digging over the weekend and let you know what i find.

Bring the manual or download a copy, and see where that gets you.

In particular, pay attention to the detailed specs of all of the equipment used (not certain of the physical implementation), but assuming the bare minimum (just system NIC ports on the switch), make sure the NIC ports are both 802.3ad and 802.1AX compliant as a start).

If you're not sure what I'm talking about, take a look at Limitations section of wiki's page on Ethernet aggregation, and see if that helps.

Good luck.

 

[1nfr4r3d]

In particular, pay attention to the detailed specs of all of the equipment used (not certain of the physical implementation), but assuming the bare minimum (just system NIC ports on the switch), make sure the NIC ports are both 802.3ad and 802.1AX compliant as a start).

hiya, ok i downloaded a manual for the 2716, it looks like this model came out before the 802.1AX specs were implemented (circa 2006) but it does support the 802.3ad spec.

I've been trying to find out the exact specs of the NICs on the mac pro but am having a time finding anything which provides significant detail, the mac profiler doesn't give much and all i can find online is:

The system has "two independent 10/100/1000BASE-T [Gigabit] Ethernet (RJ-45) interfaces with support for jumbo frames."

Does anyone know where/how i can find out more detailed spec on the NICs in the mac?

 

[funkahdafi]

hiya, ok i downloaded a manual for the 2716, it looks like this model came out before the 802.1AX specs were implemented (circa 2006) but it does support the 802.3ad spec.

AFAIK 803.3ad is all you need. I don't think Mac OS does 802.1AX at all (regarding to the online help in OS X it only uses 802.3ad).

If the switch dates back to 2006, have you tried updating the firmware? Maybe it's just a bug.

BTW: The HP Procurve switches are very cheap (I paid 180 bucks for a managed 24 port gigabit switch) and handle 802.3ad pretty well.

 

[Spacedust]

I did that on Windows 7 on my Mac Pro, so now I have 2 Gbps to my Time Capsule

You just need the latest drivers for Intel wired networking.

 

[funkahdafi]

I did that on Windows 7 on my Mac Pro, so now I have 2 Gbps to my Time Capsule

You just need the latest drivers for Intel wired networking.

Please don't confuse people. You need drivers on Windows maybe, but you certainly don't on Mac OS X!

 

[nanofrog]

hiya, ok i downloaded a manual for the 2716, it looks like this model came out before the 802.1AX specs were implemented (circa 2006) but it does support the 802.3ad spec.

802.1AX is the security layer, and it wasn't released until Nov. 3, 2008 (they found a problem with 802.1X, and had to make a correction). If you're interested in seeing the history of what/when different parts of the Aggregation specs came out, take a look here.

I made a quick check on Dell's Support site to see if newer firmware is available, and all I could find is A031.0.1.07, which is from Nov. 30, 2005 (here).

Now this doesn't mean it's impossible to get your current equipment working, but I'm not sure of if a necessary spec is missing due to age (i.e. supported on the NIC's, but not the switch), what equipment you have beyond the switch model listed (including model/year of each computer involved), firmware settings in the switch, ... that could be the cause of your problems (assuming all of your existing equipment will actually work together).

BTW, did you verify all links really are duplex?

I've been trying to find out the exact specs of the NICs on the mac pro but am having a time finding anything which provides significant detail...

Does anyone know where/how i can find out more detailed spec on the NICs in the mac?

I'm not sure if there's a specifications sheet that will provide that or not.

So you may have to get inside each computer and find the Ethernet controller chip, write down the information printed on it, and search to get more detailed information. Not the most fun thing to do, but you may not have a choice to be sure of what is/is not supported with certainty.

AFAIK 803.3ad is all you need. I don't think Mac OS does 802.1AX at all (regarding to the online help in OS X it only uses 802.3ad).

I'm still thinking it's an aggregation mismatch issue though.

Aggregation mismatch refers to not matching the aggregation type on both ends of the link. Some switches do not implement the 802.1AX standard but support static configuration of link aggregation. Therefore link aggregation between similarly statically configured switches will work, but will fail between a statically configured switch and a device that is configured for LACP.

Source.

I'm still getting the impression that this is the type of issue involved, particularly as it's pre-802.1AX compliant (I suspect your HP ProCurve is = no problems).

If the switch dates back to 2006, have you tried updating the firmware? Maybe it's just a bug.

Unfortunately there isn't.

BTW: The HP Procurve switches are very cheap (I paid 180 bucks for a managed 24 port gigabit switch) and handle 802.3ad pretty well.

Is it the HP 1810-24G ?

I ask, as there's been a price hike since late 2010, and the cheapest I can find it now is $265USD(here; never done business with the seller). Still a good price for what it does though.

 

[1nfr4r3d]

yeah one of the first things i tried looking at was the firmware, and (unfortunately?) it is up to date so ruled that one out.

all the links are full duplex, i can manually set them on the switch as well as the mac, should flow control be enabled? i had it enabled on both the switch and the mac so they would match.

the Mac i'm trying to set it up on is a a MacPro 3,1 which is from 2008 model, quad core intel xeon, 2.8GHz, 2GB RAM. If i'm honest i don't think its the mac. interestingly enough, i have a Mac Pro 1,1 which is from 2006 which i'm going to try setting up the aggregation on to see if it makes any difference seeing as its from roughly the same time period as the switch (don't know why i didn't think of that before!?)

what is interesting nanofrog is your last comment about aggregation mismatch, we know the switch doesn't support 802.1AX and there is nothing about LACP in the managed interface and the options on the mac don't refer to anything like that either - although this sounds like it could be the problem.

i do get the green lights for all of about 90 seconds before dropping, interestingly enough i measured the speed while it was "working" and it's not as fast as it is using it unmanaged on a single port, i get around 85MB/s while its working versus around 117MB/s using single port unmanaged??

 

[nanofrog]

all the links are full duplex, i can manually set them on the switch as well as the mac, should flow control be enabled? i had it enabled on both the switch and the mac so they would match.

Disable Flow Control on all devices, and see where that gets you.

The reasoning for this, is 802.1Qbb was initiated in 2008, and approved in June of this year, and the 2008 MP *could* be using an Ethernet chip that supports an early draft of it. I rather doubt this, but it's worth a shot as a full testing procedure for finding your problem (assuming all of your existing equipment is truly compatible). It should also be a quick test, so I think it worth the little bit of time it would take to investigate.

the Mac I'm trying to set it up on is a a MacPro 3,1 which is from 2008 model, quad core intel xeon, 2.8GHz, 2GB RAM. If I'm honest i don't think its the mac. interestingly enough, i have a Mac Pro 1,1 which is from 2006 which I'm going to try setting up the aggregation on to see if it makes any difference seeing as its from roughly the same time period as the switch (don't know why i didn't think of that before!?)

Worth testing.

Considering the line of thought I've been using (802.1AX), is that the MP supports it, while the switch doesn't (and the user doesn't have the ability to disable it in the MP's NIC, and force the older standard - this is an assumption based on what information you've given me; now if you're willing to write a custom driver, this could be possible).

what is interesting nanofrog is your last comment about aggregation mismatch, we know the switch doesn't support 802.1AX and there is nothing about LACP in the managed interface and the options on the mac don't refer to anything like that either - although this sounds like it could be the problem.

i do get the green lights for all of about 90 seconds before dropping, interestingly enough i measured the speed while it was "working" and it's not as fast as it is using it unmanaged on a single port, i get around 85MB/s while its working versus around 117MB/s using single port unmanaged??

I really do think it's centered around 802.1AX, and what you've gotten out of it isn't running correctly (should see close to double the performance of a single link out of it).

Ultimately, I do think you're going to need to buy a new switch (incompatibility issue due to it's age), of which the HP 1810 series would be a product you might want to take a close look at ($265 - 370 or so for a 24 port version <HP 1810-24G> when I checked yesterday).

 

[1nfr4r3d]

thanks nanofrog, i'll try those things out and let you know where i get to with it all. really appreciate your time and effort with all this.

also, while i'm in the thinking mood... i have each mac in the office setup as manual ip addresses, 192.168.1.10 - 20 - 30 etc including the server, should i be using DHPC?

 

[derbothaus]

should i be using DHPC?

Whenever possible. I do. Unless you need to have the statics for port forwards or management or reservation tables.

 

[nanofrog]

thanks nanofrog, I'll try those things out and let you know where i get to with it all. really appreciate your time and effort with all this.

NP.

also, while I'm in the thinking mood... i have each mac in the office setup as manual IP addresses, 192.168.1.10 - 20 - 30 etc including the server, should i be using DHPC?

Unless there's a specific reason you've not stated that you require static IP's, then Yes.

Like derbothaus, I prefer using DHCP whenever possible.

 

[dyn]

Whenever possible. I do. Unless you need to have the statics for port forwards or management or reservation tables.

Then you can use DHCP too. DHCP is a way of managing your ip-addresses and ip-settings, nothing more. If you need a static ip-address you simply tie it to a MAC-address (or even multiple ones). Which is why I'm always using DHCP. Most devices are set up that way by default anyway which means that I can have it working with the correct settings immediately after plugging it in. This also prevents it from getting the wrong ip-address if somebody or something manages to clear the settings and switch it back to DHCP.

 

[The Mac DDS]

Hi Guys,

has anyone managed to successfully set up an aggregated ethernet link on a Mac Pro?

I am running a server which i'm trying to set this up on and have a Dell PowerConnect 2716 which i can manage and login to via browser and set up a link aggregated group by combining ports 1 + 2. So far so good.

I then go through the network preference pane on the Mac Pro and set up a virtual interface combining the 2 ethernet ports into 1 bond / LAG. This works for all of about 90 secs then drops the connection saying the port on the switch doesn't have 802.3ad link aggregation enabled???

Does anyone have any suggestions what i can try as i am fast running out of ideas

Dropping in late to the game. Sorry for the aside, but does OS X Lion feature link aggregation or is it just a produce of Lion Server?

 

[1nfr4r3d]

i would imagine its available in both version of lion, server and standard as it is available in both versions of snow leopard.

In snow leopard you go to System Preferences > Network and at the bottom of the left pane there's a cog icon, click that and select the option that reads "Manage Virtual Interfaces".

You then get another drop down, which you click the "plus" button and choose to setup a "New Link Aggregate", add your ports and away you go... provided your setup supports it

 

[The Mac DDS]

i would imagine its available in both version of lion, server and standard as it is available in both versions of snow leopard.

In snow leopard you go to System Preferences > Network and at the bottom of the left pane there's a cog icon, click that and select the option that reads "Manage Virtual Interfaces".

You then get another drop down, which you click the "plus" button and choose to setup a "New Link Aggregate", add your ports and away you go... provided your setup supports it

Thanks, 1nfr4r3d! Yes, the steps seem to be identical in Lion. Never having done this before, I'm assuming that since my Mac Pro came with two ethernet ports, LAG is possible. I'm also assuming that LAG has to be configured both on the Mac Pro/OS level via the System Preferences settings above, and also at the switch end. I'll be using a Cisco SF 300-20 Managed Switch. I've yet to "install it", but apparently it is capable of LAG via it's web browser interface.

Cisco SF 300 Data Sheet

 

[nanofrog]

Thanks, 1nfr4r3d! Yes, the steps seem to be identical in Lion. Never having done this before, I'm assuming that since my Mac Pro came with two Ethernet ports, LAG is possible. I'm also assuming that LAG has to be configured both on the Mac Pro/OS level via the System Preferences settings above, and also at the switch end. I'll be using a Cisco SF 300-20 Managed Switch. I've yet to "install it", but apparently it is capable of LAG via it's web browser interface.

Cisco SF 300 Data Sheet

I don't see any reason it shouldn't, as its not a consumer model (know the 3500 <EOL now> and 4500 series do).