Mellanox switch Link Aggregation (bonding)

Bonding / Link Aggregation must be configured on the computer itself (this example shows Linux) and at the switch.

First configure the computer side for bonding (see section 2), then configure the switch.  The switch is shown first here as the switch is the primary objective for this article.

1) On the Mellanox switch.

1.1) IP connection note.

Web browser to the switch IP and login, default credential is: Account: admin, Password: admin.

If only using just the IP for the 1st time, you will get site can't be reached error: ERR_CONNECTION_REFUSED

1st time connecting on a particular machine must also use https://, you will get a message of connection is not private NET::ERR_CERT_AUTHORITY_INVALID error, then proceed in unsafe mode

1.2) LAG (Link Aggregation Group) creation.

ETH Mgmt > Link Aggregation.  Check the LACP Enable box.

Create LAG interfaces - In the Create LAG section, fill-in LAG Interface Number, 1 is the 1st and Apply.  Name will be created as Po1, Po2....

Click on the "Ports" tab, the LAG Group shows up as "Port channels" as shown below, 1 = Po1, 2 = Po2, etc.....

Select one of the "Port channel:"  (this is NOT a physical port # but the LAG interface, Emllanox calls this "Port channel" and it is confusing!   

Scroll down to the Port Configuration and adjust.  See Po1 (Port channel) example below.

1. The MTU to match the port's MTU (9216). 
2. FlowControl Mode to Global.
3. Click Apply

At the "Ports" tab and select the port to assign the bond to, under the Port Configuration section.  See Port 1 example below.

1. Set MTU to 9216
2. Set FlowControl Mode to Global.
3. Set LAG/MLAG to the LAG interface name created above (Po1 or Po2...) where this port should be bonded to.
4. Set LAG Mode to LACP Active.
5. Click Apply

Repeat this for each port and bond them to the appropriate LAG interface.

If setup properly these ports turns green:

ETH Mgmt will show the LAG Members:

Port 1 example:

Po1 (Port channel) example:

 

2) On the Linux computers:

These files typically exists in /etc/sysconfig/network-scripts

 bond0 is the "master" of the bond (create if done not exists):

# cat ifcfg-bond0
DEVICE=bond0
NAME=bond0
BOOTPROTO=none
MTU=9000
ONBOOT=yes
IPADDR=10.241.243.74
NETMASK=255.255.255.192
GATEWAY=10.241.243.65
NM_CONTROLLED=no
BONDING_OPTS="mode=802.3ad miimon=100 lacp_rate=fast xmit_hash_policy=layer3+4"

 ens1f0 is one of the two physical port as a "slave" to bond0:

# cat ifcfg-ens1f0
DEVICE=ens1f0
NAME=ens1f0
BOOTPROTO=none
MTU=9000
ONBOOT=yes
SLAVE=yes
MASTER=bond0
NM_CONTROLLED=no

 ens1f1 is the other physical port also as a "slave" to bond0:

# cat ifcfg-ens1f1
DEVICE=ens1f1
NAME=ens1f1
BOOTPROTO=none
MTU=9000
ONBOOT=yes
SLAVE=yes
MASTER=bond0
NM_CONTROLLED=no