bondx: an illegal loopback occurred on adapter (swpx)


Userlevel 1
After upgrading to Cumulus VX 3.2, I noticed that sometimes I'll get the following infinite looping error when adding a bond to a bridge. The error is listed for all swps that are bond-slaves. It doesn't happen all the time, however I think it may be dependent on a certain order in which things are configured.
Has anyone encountered this after adding the bond and executing an ifreload -a? Thanks.
[14584.101977] Check the configuration to verify that all adapters are connected to 802.3ad compliant switch ports
[14620.338176] bondx: An illegal loopback occurred on adapter (swpx)

9 replies

Userlevel 2
I may have taken a different route than you, and I used NCLU instead of using VI, but I didn't run into the error.

edit... tried to cut-n-paste VX output and it looked terrible. Below is the summary.

net add bond bond-b-test bond-slaves swp1
net add vlan 100
net add int bond-b-test bridge-access 100

cumulus@leaf01:/etc/snmp$ sudo ifreload -a
cumulus@leaf01:
Userlevel 2
Resulting configuration:

interface bond-b-test
bond-slaves swp1
bridge-access 100

interface bridge
bridge-ports bond-b-test
bridge-vids 100
bridge-vlan-aware yes
Hi Kevin,

I seem to be running into the same issue as "B".

I've tried a spine/leaf config on both v3.1.1 and 3.2, using both direct modification of the /etc/network/interfaces and the new NCLU commands you mentioned above. See a side by side screenshot below for the new NCLU commands on a 3.2 config:

,

I'm running a spine/leaf topology simulation via GNS3, using different procedures to enable the bond, but I'm not having much luck.

It seems like RHEL documented this issue previously, but I'm sure Cumulus VX switchports are 802.3ad compliant.



I set up an alternate, more simplistic simulation several weeks ago, pictured below, and didnt/dont have any issues with bonding interfaces there.



Thoughts?
Userlevel 2
Christopher Vanik wrote:

Hi Kevin,

I seem to be running into the same issue as "B".

I've tried a spine/leaf config on ...

I'm just ran through the same command sequence that you have and couldn't replicate on vagrant/virtualbox.

I did have one difference in my config.
Is there any reason you're sourcing /etc/network/interfaces.d/*.intf?
Christopher Vanik wrote:

Hi Kevin,

I seem to be running into the same issue as "B".

I've tried a spine/leaf config on ...

Hm, well I'm not explicitly sourcing the file from my end, so no - I guess I assumed that was being done as part of the net commit command. If its not needed, I can manually remove it.
Userlevel 3
The only bonding mode supported is 802.3ad. Please provide the following output from both sides of the bond:
cat /proc/net/bonding/bond0
If I recall, the illegal loopback is due to the LACP PDUs arriving on each member link with mismatched information. This appears as a problem, since LACP expects all member links to be connected between the same 2 devices.

If possible, also provide the full configuration of all switches participating in the bond, so we can try and reproduce this.

Jason Guy wrote:

The only bonding mode supported is 802.3ad. Please provide the following output from both sides o...

Sure thing, please see the full config & bond status for both my spine & leaf switches listed below.

LEAF:


root@cumulus:~# cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*.intf

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet dhcp

auto swp1
iface swp1

auto swp2
iface swp2

auto bond0
iface bond0
bond-slaves swp1 swp2
bridge-access 100

auto bridge
iface bridge
bridge-ports bond0
bridge-vids 100
bridge-vlan-aware yes

root@cumulus:~# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: down
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: fast
Min links: 1
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 08:00:27:18:ce:9c
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 1
Actor Key: 9
Partner Key: 1
Partner Mac Address: 00:00:00:00:00:00

Slave Interface: swp2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:18:ce:9c
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: churned
Actor Churned Count: 0
Partner Churned Count: 1
details actor lacp pdu:
system priority: 65535
system mac address: 08:00:27:18:ce:9c
port key: 9
port priority: 255
port number: 1
port state: 79
details partner lacp pdu:
system priority: 65535
system mac address: 00:00:00:00:00:00
oper key: 1
port priority: 255
port number: 1
port state: 1

Slave Interface: swp1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27🆎49:f6
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: churned
Partner Churn State: churned
Actor Churned Count: 1
Partner Churned Count: 1
details actor lacp pdu:
system priority: 65535
system mac address: 08:00:27:18:ce:9c
port key: 9
port priority: 255
port number: 2
port state: 71
details partner lacp pdu:
system priority: 65535
system mac address: 00:00:00:00:00:00
oper key: 1
port priority: 255
port number: 1
port state: 1

SPINE:

cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*.intf

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet dhcp

auto swp1
iface swp1

auto swp2
iface swp2

auto bond0
iface bond0
bond-slaves swp1 swp2
root@cumulus:~# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: down
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: fast
Min links: 1
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 08:00:27:18:ce:9c
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 1
Actor Key: 9
Partner Key: 1
Partner Mac Address: 00:00:00:00:00:00

Slave Interface: swp2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:18:ce:9c
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: churned
Actor Churned Count: 0
Partner Churned Count: 1
details actor lacp pdu:
system priority: 65535
system mac address: 08:00:27:18:ce:9c
port key: 9
port priority: 255
port number: 1
port state: 79
details partner lacp pdu:
system priority: 65535
system mac address: 00:00:00:00:00:00
oper key: 1
port priority: 255
port number: 1
port state: 1

Slave Interface: swp1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27🆎49:f6
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: churned
Partner Churn State: churned
Actor Churned Count: 1
Partner Churned Count: 1
details actor lacp pdu:
system priority: 65535
system mac address: 08:00:27:18:ce:9c
port key: 9
port priority: 255
port number: 2
port state: 71
details partner lacp pdu:
system priority: 65535
system mac address: 00:00:00:00:00:00
oper key: 1
port priority: 255
port number: 1
port state: 1
root@cumulus:~#

Userlevel 3
Jason Guy wrote:

The only bonding mode supported is 802.3ad. Please provide the following output from both sides o...

Hi Christopher,

Looks like you have the same mac address on both sides of the member?
Leaf -
swp1: 08:00:27🆎49:f6
swp2: 08:00:27:18:ce:9c

Spine -
swp1: 08:00:27🆎49:f6
swp2: 08:00:27:18:ce:9c

Assuming swp1's are connected, and swp2's are connected, then the LACP would have a problem with this. If this is VX, make sure to set the interface MAC's to be different.

Jason Guy wrote:

The only bonding mode supported is 802.3ad. Please provide the following output from both sides o...

Whoops, I feel silly. I thought I had reinitialized my MACs when I imported the VX, but apparently it didnt take or something.

I cycled the addresses on each virtual adapter and the problem seems to have been resolved. Thanks so much Jason!

Reply