LAG Question

[mvmanolov]

So, I've tried a number of things and for the life of me can't get more than gigabit speeds 100 (or so) MB/s through my LAG.

Set up: Mac Mini (Server) (with a 256 Samsung 830 SSD - so not the bottleneck)
Both Ethernet and TB Ethernet adapters of the mini are connected to my switch (NetGear GST716T v.2) over Cat6a cable the ports on the switch for these two are configured as one LAG (Lag 1) and the mini is configured in LAG and the OsX bond is working fine from all i can tell, same on the switch.

I tried to use this LAG for 2 tests.

Test 1:
i connected a MBP both through Ethernet and TB again with two Cat6a cable to two other ports on the switch and created another LAG (Lag 2). the OsX bond worked fine. but when i tried to pull a large (10+GB) file from the mini i was only getting the 100+MB/s.

So then i tried to configure all four ports on the switch in a single LAG (Lag 1) (as i thought that this may be the issue). when i did that, the OsX bond did not work - the status lights in net preferences were both red and it wouldn't get an IP from the DHCP server, i tried to rebuild the bond from scratch and still it did not work - same outcome.

Test 2:
Then I connected the MBP via Ethernet and a MBA via TB to the existing 4 port LAG on the switch. Both laptops were successfully connected to the LAN but when i tried to pull large files from the mini to each of the laptops simultaneously i was again getting only 100+MB/s combined (so 50-60 each).

Then i disconnected the 2 non-mini ports on the switch from the LAG (Lag1 - which now only included the two mini ports) and tried to transfer again with the same result.


Here is what i understand to be possible (and what i ultimately want to be able to do) through a LAG: The file server on the LAG should be able to send a large file to each individual computer connected to it at the full gigabit rate (100+MB/s) for a combined total of 200+MB/s. (If both clients are downloading simultaneously.)

If i am mistaken about my understanding please correct me. and Let me know how this can be accomplished otherwise than through a LAG.

If i am not mistaken, then can someone tell me what i am doing wrong in setting up my LAG?

Thank you.
M,

 

[AmestrisXServe]

So, I've tried a number of things and for the life of me can't get more than gigabit speeds 100 (or so) MB/s through my LAG.

Set up: Mac Mini (Server) (with a 256 Samsung 830 SSD - so not the bottleneck)
Both Ethernet and TB Ethernet adapters of the mini are connected to my switch (NetGear GST716T v.2) over Cat6a cable the ports on the switch for these two are configured as one LAG (Lag 1) and the mini is configured in LAG and the OsX bond is working fine from all i can tell, same on the switch.

I tried to use this LAG for 2 tests.

Test 1:
i connected a MBP both through Ethernet and TB again with two Cat6a cable to two other ports on the switch and created another LAG (Lag 2). the OsX bond worked fine. but when i tried to pull a large (10+GB) file from the mini i was only getting the 100+MB/s.

So then i tried to configure all four ports on the switch in a single LAG (Lag 1) (as i thought that this may be the issue). when i did that, the OsX bond did not work - the status lights in net preferences were both red and it wouldn't get an IP from the DHCP server, i tried to rebuild the bond from scratch and still it did not work - same outcome.

Test 2:
Then I connected the MBP via Ethernet and a MBA via TB to the existing 4 port LAG on the switch. Both laptops were successfully connected to the LAN but when i tried to pull large files from the mini to each of the laptops simultaneously i was again getting only 100+MB/s combined (so 50-60 each).

Then i disconnected the 2 non-mini ports on the switch from the LAG (Lag1 - which now only included the two mini ports) and tried to transfer again with the same result.


Here is what i understand to be possible (and what i ultimately want to be able to do) through a LAG: The file server on the LAG should be able to send a large file to each individual computer connected to it at the full gigabit rate (100+MB/s) for a combined total of 200+MB/s. (If both clients are downloading simultaneously.)

If i am mistaken about my understanding please correct me. and Let me know how this can be accomplished otherwise than through a LAG.

If i am not mistaken, then can someone tell me what i am doing wrong in setting up my LAG?

Thank you.
M,

You are correct, in that a LAG should operate in 'parallel' operation. I haven't done much with Thunderbolt, so it could be a problem with the TB host and your NIC.

What I suspect is that you have created a bond, but your connection is still using eth0 or eth1 for the data transfer instead of bond0 on at least one component of your LAN.

Are you certain that you are using the bond0 interface, and not one of the ethX interfaces to do your data transfer? You should do a test over bond0 in a shell with iperf (available via MacPorts), or a similar tool.

You should check this first, before going much further, and check it on every side of the connection where you have a LAG.

Useful: You may also want to try a bond0 test with the two systems, without the switch, using crossover cables, (1:1 directly). That could help rule out if the problem is the switch, and any protocols that it uses that aren't supported, or properly configured in OSX, and will allow you to test the bond0 connections and so forth on each system directly, reducing the factors in determining the source of your problem.

Questions, Advice, and Pointers

How exactly have you added a second Ethernet NIC to the MacBook Pro in your LAN, and what NIC did you use? What MBP model and what Mini model are you using?

If I am reading this right, your Mini is connected to a TB cage with a PCIe NIC. If so, what NIC are you using on your TB PCIe cage?

The switch you are using may not be properly set up: Have you done the tests?
Read: http://www.downloads.netgear.com/files/GDC/FS728TS/fs700ts_configuring_LAG.pdf

If the SSD inside the Mini, or connected via TB as DAS?

What version of OSX are you running?
You may find something in these threads helpful, bearing in mind that my OSX expertise runs through 10.6. It isn't going to be 100% topical, but I see problems in other areas in trying to enable a LAG on MacBook and Mini systems.

https://discussions.apple.com/message/20937215#20937215
https://discussions.apple.com/thread/4318748
https://discussions.apple.com/message/7351506#7351506

A previous, similar thread here:
https://forums.macrumors.com/threads/1516075/

What drive is in the MacBook pro?

To be frank, you may be better off making a Thunderbolt interface between the two systems, or through a switch--I think there are good TB switches now--as each TB controller should support five devices, and you can also LAG those in the future if you need obscene speeds that your drives will never support.
https://forums.macrumors.com/threads/1657957/

How many clients do you anticipate using the LAG on your network? You will need to configure it on each, so you may want to consider a netboot image to deploy the OS to your clients with a preconfigured connection, which may save you the headache of adding a LAG interface on client systems.

Those are my thoughts for the present.

 

[chrfr]

using crossover cables

Crossover cables aren't necessary with Mac ethernet hardware anymore, if the Mac is equipped with gigabit ethernet.

If I am reading this right, your Mini is connected to a TB cage with a PCIe NIC.

Thunderbolt to ethernet adapters exist. http://store.apple.com/us/product/MD463ZM/A/thunderbolt-to-gigabit-ethernet-adapter

 

[AmestrisXServe]

Crossover cables aren't necessary with Mac ethernet hardware anymore, if the Mac is equipped with gigabit ethernet.

I suppose not with the newer systems, but I always use a cross cable to connect two systems, ignoring AMDIX, as I am rather paranoid. :/

I rarely need to connect two mac products together anyway. It's usually switches, or on occasion, two non-conforming systems. I have a few in a drawer in a server cabinet handy, so I never need to worry about if I have them on-site, and I don't bother using standard Cat-6, as I want to be able to clearly rule out any possibility of cabling problems beforehand.

I find it eases diagnostic flows when you don't have too many possible causes for a fault, so why start with a cable that may, or may not work?

I also find it good to maintain use of cross cables when you link two systems without a switch, to keep it in your head that you need them, even if you don't, because if you get too accustomed to AMDIX, you might slip up at some point because you rely on it being implemented, when it is not. I think my network is a 40/60 split between supporting and not supporting it, with not supporting at a conservative 60%.

In this instance however, you are correct. If the OP doesn't have cross cables, he can use standard Cat-6 to connect two modern Macs. I am a bit surprised by the model list, as I thought that the AMDIX implementation started a little later than it did. It seems spotty right around the iBook era, as contemporary systems from 2002-2004 may or may not support AMDIX.

Most of my G4 systems do need cross cables, but a few might not, from the list, which is not 100% model specific. (i.e. An iBook and a Powerbook G4 Titanium from the same year do not both support AMDIX; but do all Powerbook G4 models support it, or just the DVI and newer models?)

I don't know at exactly what point Apple decided to include it with every Mac, but I would rather not gamble on it.

Thunderbolt to ethernet adapters exist. http://store.apple.com/us/product/MD463ZM/A/thunderbolt-to-gigabit-ethernet-adapter

Right, but I doubt that all of those adapters support 802.3ad and 802.1ax. I would further be worried which protocol the switch is using, compared to the client and host systems, if LACP, or pinning is involved, and if anything is using FLAG. (FLAG can also cause all sorts of headaches.)

The question I asked, is relevant whether the OP is using a PCIe cage with a NIC in it, or an adapter module, but I would always advise on the former, rather than the latter, as you have more reliable information (data sheets), and because you can ensure that your second NIC uses the same host adapter, firmware, and whatnot as your main NIC.

Trying to bind two NICs that are using different protocols, or trying to use FLAG, are both going to result in a problem of data splitting, broken bonding, and reception end bandwidth. (For the record, 802.1ax is the way to go, whenever possible.)

That is why I suggest using the two Mac systems in a single LAN (1:1) to test the binding, and the bond0 interface, as the problem could be that the systems are suing a protocol that differs from that supported by the switch. I haven't read the data sheet yet to find out which IEEE set it uses.

The model number in the OP is invalid, but I am guessing it is a Netgear GS7-16T. I usually go with Cisco stuff, as they usually include all protocols, including their own proprietary stuff, whereas other switches won't have all of the Cisco-specific protocols to communicate in series.

The Netgear unit looks decent, but until i read a data sheet for it, I won't be certain. I still advise connecting the two systems to each-other without the switch, to check the bonded interface.

At least in doing so, you can rule out if the switch is causing the trouble, or if it one of the two systems (or possibly both), whether that is due to software or hardware. (I really need to know what NICs are in use to diagnose any HW problem on the systems.)

 

[chrfr]

I suppose not with the newer systems, but I always use a cross cable to connect two systems, ignoring AMDIX, as I am rather paranoid. :/

Auto MDI-X is part of the gigabit standard so it will work with any modern computer.
If only there was a technical document for which Macs required crossover cables...
http://support.apple.com/kb/ht2274

I rarely need to connect two mac products together anyway.

Only one device needs to support AMDI-X.

 

[AmestrisXServe]

Auto MDI-X is part of the gigabit standard so it will work with any modern computer.
If only there was a technical document for which Macs required crossover cables...
http://support.apple.com/kb/ht2274


Only one device needs to support AMDI-X.

I've read that page: It lists systems by grouping, not by model. For example, where does a M9290LL/A (a 20" iMac G4 with a rotating display) fit into the picture? It isn't listed either in any group, or by model.

You have to drop to a spec sheet to check, or remember that list and trust that either you, or it are not in error, which in my opinion, just wastes time, when it takes all of forty seconds to grab a cross cable, which assures you that the link will work.

For the record, even if one device has AMDIX, if the other is MDIX, some connections may not be enabled for autonegotiation, and in a hybrid network where not every system is Gigabit enabled, I would rather trust to the old-fashioned approach for surety. Why is that so bloody hard to understand? Why are we even debating this?

Why are you critiquing my workflow methodology, instead of assisting the OP with his problem?

I am already seeing a lot of that kind of narcissism here: Why are members performing autopsies on replies to a post, rather than responding to the post itself? Your input would be much more valuable if it addressed why the OP isn't able to create his LAG, rather than nitpicking about a bleedin' cable type. I honestly don't feel like debating something that petty: It was one thing to point out that the OP doesn't need a cross cable for his connection, but it's another to sidetrack into my lack of necessity. I already explained my reasons, and if you don't believe they are valid, jolly good for you, as you don't have to do my work for me.

I have to be 100% certain of what I am doing, when I am doing it, with no room for negotiation or error, so I use what I know will work no-matter what types of connections I am using. Isn't that straightforward enough to understand?

Before you ask, I can tell each cable type at a glance by colour. All of my cross cables are orange, all my Cat-5e are grey, and any other colour is Cat-6, except for violet, which is Cat-6S, and rarely needed. That means that if i need a Cat-6S or cxross cable, they stand out as violet and orange, and I know if I see a grey gable that it is Cat-5e and shouldn't be used for Gigabit connections. I like simple, easy for me to remember, rulesets to prevent errors.

Beyond that I use colour coding for servers in racks:
Blue => OSX; and front-panel use for bridging switch patches. (Blue, for the old Mac Face logo and Bondi)
Yellow => Windows NT (Yellow, for ..you can guess.)
Red => Linux (Red for 'Red Hat')
Green => Storage Device or Router/Gateway (Green for 'G'ateway)
White => Solaris/Unix/Other (White, for 'Sun' light Spectrum), and the IP phone in the cabinet.
Black => Something very expensive, or critical (black gambling marker), or custom HW peripherals.

I use Grey for Cat-5e = Grey, as 'Old and Grey'.
Violet for Shielded: 'Purple haze', 'fog', 'covered'.
Orange for Cross: 'Combining Apples and Oranges'

It's all part of a memory system; going senile isn't very fun, and my short-term memory is horrible at this point, so to tell something at a glance, increasing speed and productivity, I have q variety of OCD-ish systems like this for my work.

For client systems, I generally don't need to bother about cable colour, so I use blue or white cables for Cat-6 on clients, and grey for Cat-5e.

 

[mvmanolov]

i am quite busy this weekend.. have a project to finish for wednesday/thursday so i'll reply to all questions and provide details about the setup after that.

Thank you for taking the time!

I really appreciate it!
M,

 

[mvmanolov]

Solved!

HA!
I am a genius... ok perhaps not...but i solved my problem and am now a proud owner of my very own fully functional 220+MB/s LAG

So here is what was the problem:

The mini functions as the file server! so i have the router assign it one static IP on the basis of the MAC of the ethernet connection. and a separate static IP for the Thunderbolt-ethernet connection.

so when i originally created the LAG it automatically would assign the IP address of the Thunderbolt-ethernet connection. However that was a problem for me as all my services are configured to run under the Ethernet IP, so i changed the IP of the LAG manually to the one i want (so the Ethernet one rather than the thunderbolt-ethernet one).

And it was precisely this manual change that for some reason unknown to me was making the LAG work only at half speed!

since i figured that out... i changed the DHCP reservation so that the TB-ETH ip is the former ETH IP and vice-versa. So now when the LAG bond in OsX automatically assigns the IP it's the one i want and the LAG is fully functional

I am quite proud that i got it working, but don't understand why manually assigning the ETH IP to the bond breaks it...So if someone whats to take a crack at explaining i'd very much appreciate the learning

In either case though, i wanted to say a huge thanks to both of you for taking the time to post to this and give me such wonderful, detailed and educational advice!

So BIIIGGG THANKS

 

[AmestrisXServe]

HA!
I am a genius... ok perhaps not...but i solved my problem and am now a proud owner of my very own fully functional 220+MB/s LAG

So here is what was the problem:

The mini functions as the file server! so i have the router assign it one static IP on the basis of the MAC of the ethernet connection. and a separate static IP for the Thunderbolt-ethernet connection.

so when i originally created the LAG it automatically would assign the IP address of the Thunderbolt-ethernet connection. However that was a problem for me as all my services are configured to run under the Ethernet IP, so i changed the IP of the LAG manually to the one i want (so the Ethernet one rather than the thunderbolt-ethernet one).

And it was precisely this manual change that for some reason unknown to me was making the LAG work only at half speed!

since i figured that out... i changed the DHCP reservation so that the TB-ETH ip is the former ETH IP and vice-versa. So now when the LAG bond in OsX automatically assigns the IP it's the one i want and the LAG is fully functional

I am quite proud that i got it working, but don't understand why manually assigning the ETH IP to the bond breaks it...So if someone whats to take a crack at explaining i'd very much appreciate the learning

In either case though, i wanted to say a huge thanks to both of you for taking the time to post to this and give me such wonderful, detailed and educational advice!

So BIIIGGG THANKS

What is the output of ifconfig bond0? (Replace bond0 with the interface name of your bonded connection interface, if it differs from bond0.)

Did you change the IP on the interface on the Mac, and not re-assign the IP settings on your switch?

I expect that when you created the bond, the binding was assigned an IP based on part of the original interface on your server, so when you changed it, on your switch, you essentially overrode the bond. MacOS X doesn't handle binding very well in my experience, and is riddled with problems.

I assume that you have changed the IPs for your service resolution/NAT, etc..

It is easily possible that this was either a problem with the bond configuration on your Mac system, or with the switch configuration. Either way, you are now up and running.