Cumulus VX support MLAG?


Hi folks,

Probably an easy question, does Cumulus VX support MLAG? I've followed numerous articles on how to set up a MLAG with Cumulus, however I am reciving the same bonding error.
Check the configuration to verify that all adapters are connected to 802.3ad compliant switch ports. bonding: peerlink: An illegal loopback occurred on adapter (swp6)
Check the configuration to verify that all adapters are connected to 802.3ad compliant switch ports. bonding: peerlink: An illegal loopback occurred on adapter (swp7)



Here is my interfaces.cf
auto swp6        iface swp6  auto swp7        iface swp7  auto peerlinkiface peerlink
bond-slaves swp6 swp7
bond-mode 802.3ad
bond-miimon 100
bond-min-links 1
bond-use-carrier 1
bond-lacp-rate 1
bond-xmit-hash-policy layer3+4
auto peerlink.4094
iface peerlink.4094
address 192.168.60.1/30
clagd-enable yes
clagd-priority 4096
clagd-peer-ip 192.168.60.2
clagd-args --vm
clagd-sys-mac 44:39:39:ff:40:94

20 replies

Userlevel 4
MLAG is a layer 2 solution, you are using layer 3 bonds. You need a bridge that contains the bonds. See Scott's walk though here:https://community.cumulusnetworks.com/cumulus/topics/spinning-up-a-virtual-mlag-environment
Userlevel 4
Sean Cavanaugh wrote:

MLAG is a layer 2 solution, you are using layer 3 bonds. You need a bridge that contains the bond...

Actually you can see an example config here:http://docs.cumulusnetworks.com/display/DOCS/Multi-Chassis+Link+Aggregation+-+MLAG
Userlevel 4
Sean Cavanaugh wrote:

MLAG is a layer 2 solution, you are using layer 3 bonds. You need a bridge that contains the bond...

Just remember to keep the clagd-args --vm with the config I linked to !
Userlevel 5
I like the example image here to show some of the hierarchy involved in the bridge and bond structure --> https://docs.cumulusnetworks.com/download/attachments/2722594/mlag_basic.png?version=1&modificationDate=1446241583000&api=v2 Basically the peerlink bond and the host-facing interfaces are all members of the bridge. Something like this: auto swp5 iface swp5 auto swp6 iface swp6 auto swp7 iface swp7 auto host1 iface host1 bond-slaves swp5 bond-mode 802.3ad bond-miimon 100 bond-lacp-rate 1 bond-min-links 1 bond-xmit-hash-policy layer3+4 clag-id 1 mstpctl-portadminedge yes mstpctl-bpduguard yes auto peerlink iface peerlink bond-slaves swp6 swp7 bond-mode 802.3ad bond-miimon 100 bond-min-links 1 bond-lacp-rate 1 bond-xmit-hash-policy layer3+4 auto peerlink.4094 iface peerlink.4094 address 192.168.60.1/30 clagd-enable yes clagd-priority 4096 clagd-peer-ip 192.168.60.2 clagd-args --vm clagd-sys-mac 44:39:39:ff:40:94 auto bridge iface bridge bridge-vlan-aware yes bridge-ports host1 peerlink bridge-stp on bridge-vids 1000-3000 # <-- adjust for whatever vlans you need bridge-pvid 1
Here is my config now, but I am still getting the loopback error
auto swp5        iface swp5
auto swp6
iface swp6
auto swp7
iface swp7
auto peerlink
iface peerlink
bond-slaves swp6 swp7
bond-mode 802.3ad
bond-miimon 100
bond-min-links 1
bond-use-carrier 1
bond-lacp-rate 1
bond-xmit-hash-policy layer3+4
auto peerlink.4094
iface peerlink.4094
address 192.168.60.1/30
clagd-enable yes
clagd-priority 4096
clagd-peer-ip 192.168.60.2
clagd-args --vm
clagd-sys-mac 44:39:39:ff:40:94
auto host1
iface host1
bond-slaves swp5
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate 1
bond-min-links 1
bond-xmit-hash-policy layer3+4
clag-id 1
mstpctl-portadminedge yes
mstpctl-bpduguard yes
auto bridge
iface bridge
bridge-vlan-aware yes
bridge-ports host1 peerlink
bridge-stp on
bridge-vids 1000-3000
bridge-pvid 1

Am I missing something else? Thank you both for your help.

Userlevel 5
Can you copy/paste the error you're seeing here? All of our configs have not included a loopback. You can try adding one to see if that helps with your error:

auto lo
iface lo inet loop

I always have a loopback (or rather, leave the existing one alone) but don't always include it in the configs for brevity. Linux in general really likes having a loopback, I can't think of a single distribution that ships without one. I'm not aware of any specific dependencies of CLAG on a loopback though. Please post the error you're seeing if you get a moment.
Sure, they are VirtualBoxes so I can provide a screenshot. I do have the auto lo/iface lo inet loopback in the interfaces.cf


Userlevel 5
This might be a factor of two different things, In Virtualbox, it is necessary to set the interfaces to promiscuous mode outside of VX in the Virtualbox interface (see this article: https://community.cumulusnetworks.com... ) in order to appropriately pass traffic through the L2 switch built into virtualbox. It is also necessary when working with bonds to include the following "ip link set swp promisc on" within the VX OS to again pass traffic properly through the hidden l2 switch. Here is my full configuration from VX 2.5.4 that replicates your setup with a few ports switched around based on what I already had wired-up:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet dhcp

# Static setup
#iface eth0 inet static
# address 10.0.2.15
# netmask 255.255.255.0
# gateway 10.0.2.2

# The contents below are automatically generated by Vagrant.
auto swp1
iface swp1
#required for traffic to flow on Bonds in Vbox VMs
post-up ip link set $IFACE promisc on

# The contents below are automatically generated by Vagrant.
auto swp2
iface swp2
#required for traffic to flow on Bonds in Vbox VMs
post-up ip link set $IFACE promisc on

# The contents below are automatically generated by Vagrant.
auto swp49
iface swp49
#required for traffic to flow on Bonds in Vbox VMs
post-up ip link set $IFACE promisc on

# The contents below are automatically generated by Vagrant.
auto swp50
iface swp50
address 192.168.1.1/24

auto peerlink
iface peerlink
bond-slaves swp49
bond-mode 802.3ad
bond-miimon 100
bond-min-links 1
bond-lacp-rate 1
bond-xmit-hash-policy layer3+4

auto peerlink.4094
iface peerlink.4094
address 192.168.60.1/30
clagd-enable yes
clagd-priority 4096
clagd-peer-ip 192.168.60.2
clagd-args ' --vm '
clagd-sys-mac 44:39:39:ff:40:94

auto host1
iface host1
bond-slaves swp1
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate 1
bond-min-links 1
bond-xmit-hash-policy layer3+4
clag-id 1
mstpctl-portadminedge yes
mstpctl-bpduguard yes

auto host2
iface host2
bond-slaves swp2
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate 1
bond-min-links 1
bond-xmit-hash-policy layer3+4
clag-id 2
mstpctl-portadminedge yes
mstpctl-bpduguard yes

auto bridge
iface bridge
bridge-vlan-aware yes
bridge-ports host1 peerlink
bridge-stp on
bridge-vids 1000-3000
bridge-pvid 1

Here is the output from clagctl: showing the CLAG/MLAG is up:

root@l1:/home/vagrant# clagctl
The peer is alive
Our Priority, ID, and Role: 4096 08:00:27:2a:b8:ab primary
Peer Priority, ID, and Role: 8192 08:00:27:0a:a0:fb secondary
Peer Interface and IP: peerlink.4094 192.168.60.2
Backup IP: (inactive)
System MAC: 44:39:39:ff:40:94

CLAG Interfaces
Our Interface Peer Interface CLAG Id Conflicts Proto-Down Reason
---------------- ---------------- ------- -------------------- -----------------
host2 host2 2 - -
host1 host1 1 - -
My VBoxManage showvminfo "LEAF1", show's the NICs are in promisc
NIC 1:           MAC: 0800272A5606, Attachment: Host-only Interface 'vboxnet0', Cable connected: on, Trace: off (file: none), Type: virtio, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: deny, Bandwidth group: none  NIC 2:           MAC: 080027F1D1CD, Attachment: Internal Network 'intnet', Cable connected: off, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: allow-all, Bandwidth group: none  NIC 3:           MAC: 0800276EF0D1, Attachment: Internal Network 'intnet', Cable connected: off, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: allow-all, Bandwidth group: none  NIC 4:           MAC: 080027F8B77F, Attachment: Internal Network 'intnet', Cable connected: on, Trace: off (file: none), Type: virtio, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: allow-all, Bandwidth group: none  NIC 5:           MAC: 08002704DCFC, Attachment: Internal Network 'intnet', Cable connected: on, Trace: off (file: none), Type: virtio, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: allow-all, Bandwidth group: none  NIC 6:           MAC: 0800273EA690, Attachment: Generic 'UDPTunnel' { dest='127.0.0.1', dport='10004', sport='10005' }, Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: allow-all, Bandwidth group: none  NIC 7:           MAC: 08002754F62E, Attachment: Generic 'UDPTunnel' { dest='127.0.0.1', dport='10001', sport='10000' }, Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: allow-all, Bandwidth group: none  NIC 8:           MAC: 0800275948B5, Attachment: Generic 'UDPTunnel' { dest='127.0.0.1', dport='10003', sport='10002' }, Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: deny, Bandwidth group: none  


Also update Interfaces.cf
auto swp1        iface swp1
post-up ip link set $IFACE promisc on
auto swp2
iface swp2
post-up ip link set $IFACE promisc on
auto swp3
iface swp3
post-up ip link set $IFACE promisc on
auto swp4
iface swp4
post-up ip link set $IFACE promisc on
auto swp5
iface swp5
post-up ip link set $IFACE promisc on
auto swp6
iface swp6
post-up ip link set $IFACE promisc on
auto swp7
iface swp7
post-up ip link set $IFACE promisc on
auto peerlink iface peerlink
bond-slaves swp6 swp7
bond-mode 802.3ad
bond-miimon 100
bond-min-links 1
bond-use-carrier 1
bond-lacp-rate 1
bond-xmit-hash-policy layer3+4
auto peerlink.4094
iface peerlink.4094
address 192.168.60.1/30
clagd-enable yes
clagd-priority 4096
clagd-peer-ip 192.168.60.2
clagd-args --vm
clagd-sys-mac 44:39:39:ff:40:94
auto host1
iface host1
bond-slaves swp5
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate 1
bond-min-links 1
bond-xmit-hash-policy layer3+4
clag-id 1
mstpctl-portadminedge yes
mstpctl-bpduguard yes
auto bridge
iface bridge
bridge-vlan-aware yes
bridge-ports host1 peerlink
bridge-stp on
bridge-vids 1000-3000
bridge-pvid 1

Where you able to get it working? Can you zip and tar.gz your gns3 topology files? I really don't know why this isn't working.

Userlevel 5
I think I know what the issue might be... try using the first few ports. In other words, use ports swp1, swp2, swp3 with no gaps in between. So I would use swp1, swp2, swp3 etc instead of swp5,6,7. There is a brief note on this in our GNS3 guide ( "e1 in GNS3 corresponds to swp1 in Cumulus VX, e2 to swp2, and so forth" https://docs.cumulusnetworks.com/display/VX/Using+GNS3+with+VirtualBox+VMs ) but it is not explained as to why this is needed. Here is the explanation (which I will seek to add to the Documentation): Essentially when an interface is added in GNS3 it is then created in the VM and in VX vms (today in VX version 2.5.4 and VX version 2.5.3), they are created in order Eth1, Eth2, Eth3 etc. So E1 in GNS3 would map to Eth1 in Vx by default and then when VX boots up, it sees an eth1 interface exists in the VM and renames that interface to SWP1, Eth2 is renamed to SWP2 and so forth. The chance to use SWP5,6,7 would arise only after already creating 7 interfaces Eth1-7 or in GNS3 terms E1-7 Once you straighten out the wiring as described above, you can double-check your port wirings with LLDP which is installed by default. Use "sudo lldpctl" on any of the VX nodes to make sure links are wired together as you are prescribing in the GNS3 topology. You had asked what I am using to get this to work. I am personally using Vagrant with the Virtualbox provider in my setup which wires the ports together using a vagrantfile instead of GNS3. I am also using some additional scripting to re-map interface names out of order. I see from your configuration that GNS3 is using the UDP tunnels as well which is really cool. I have not had a chance to play with the UDP tunnels in Virtualbox to connect VMs. It is possible you may not need all of the adjustments mentioned above for the hidden L2 switch since I believe you are bypassing that by using the UDP tunnels. Hope this helps!
Please check this post thoroughly(most importantly all the comments) for MLAG/CLAG support in Cumulus VX.

https://community.cumulusnetworks.com/cumulus/topics/need-help-with-cumulus-vx-host-multi-chassis-lag-clag

I had also posted on getting started with Cumulus VX in gns3 community.

@Eric
lldpctl shows each leaf sees the other on swp1 and 2 (I modified my topology to your recommendation of using swp1 and 2 because of GNS3 limitation. Unfortunately I am receiving the "bonding: peerlink: An illegal loopback occurred on adapter (swp2) + (swp1)" I will try to search for more anwsers, but am leaning towards this is not possible to do with Cumulus VX and GNS3

@Hemant, your post doesn't show an answer to how you got it working? even the Cumulus Rep was shocked it just started working...

Userlevel 5
Jim A wrote:

@Eric
lldpctl shows each leaf sees the other on swp1 and 2 (I modified my topology to your...

Hi, I just wanted to say Cumulus VX DOES support MLAG.

Indeed it is a mystery that how it got working, I do a couple of reboots and now it works most of the times, after 3-4 tries.
Jim A wrote:

@Eric
lldpctl shows each leaf sees the other on swp1 and 2 (I modified my topology to your...

@Eric
Thank you for taking the time to set it up. So I’ve destroyed and recreated the VXs and the GNS3 topology to closely match yours. See screen-shot below. I am still receiving the illegal loop-back message 😞 The error messages are occurring more frequently now.



LEAF1-1:
auto lo
iface lo
address 1.1.1.1/32

auto swp1
iface swp1

auto swp2
iface swp2

auto swp3
iface swp3

auto swp4
iface swp4

auto swp5
iface swp5

auto swp5
iface swp5

auto swp6
iface swp6

auto swp7
iface swp7

auto peerlink
iface peerlink
bond-mode 802.3ad
bond-slaves swp1
bond-miimon 100
bond-xmit-hash-policy layer3+4
bond-min-links 1

auto peerlink.4094
iface peerlink.4094
address 192.168.2.1/30
clagd-priority 8192
clagd-sys-mac 44:38:39:ff:00:01
clagd-peer-ip 192.168.2.2
clagd-args --vm

auto host1
iface host1
bond-mode 802.3ad
bond-slaves swp3
bond-miimon 100
bond-xmit-hash-policy layer3+4
bond-min-links 1
clag-id 1

auto bridge
iface bridge
bridge-vlan-aware yes
bridge-ports peerlink host1
bridge-vids 1-2000
bridge-pvid 1
bridge-stp on

auto bridge.1
iface bridge.1
address 192.168.1.1/24

LEAF1-1:
auto lo
iface lo
address 2.2.2.2/32

auto swp1
iface swp1

auto swp2
iface swp2

auto swp3
iface swp3

auto swp4
iface swp4

auto swp5
iface swp5

auto swp5
iface swp5

auto swp6
iface swp6

auto swp7
iface swp7

auto peerlink
iface peerlink
bond-mode 802.3ad
bond-slaves swp1
bond-miimon 100
bond-xmit-hash-policy layer3+4
bond-min-links 1

auto peerlink.4094
iface peerlink.4094
address 192.168.2.2/30
clagd-priority 8192
clagd-sys-mac 44:38:39:ff:00:01
clagd-peer-ip 192.168.2.1
clagd-args --vm

auto host1
iface host1
bond-mode 802.3ad
bond-slaves swp3
bond-miimon 100
bond-xmit-hash-policy layer3+4
bond-min-links 1
clag-id 1

auto bridge
iface bridge
bridge-vlan-aware yes
bridge-ports peerlink host1
bridge-vids 1-2000
bridge-pvid 1
bridge-stp on

auto bridge.1
iface bridge.1
address 192.168.1.2/24

Jim A wrote:

@Eric
lldpctl shows each leaf sees the other on swp1 and 2 (I modified my topology to your...

Here is VirtualBox VM templates settings in GNS3


Jim A wrote:

@Eric
lldpctl shows each leaf sees the other on swp1 and 2 (I modified my topology to your...

Example of VirtualBox NIC settings generated automatically by GNS3


Jim A wrote:

@Eric
lldpctl shows each leaf sees the other on swp1 and 2 (I modified my topology to your...

clagctl output and ping from LEAF2-1 to LEAF1-1 fails...


Userlevel 5
Jim A wrote:

@Eric
lldpctl shows each leaf sees the other on swp1 and 2 (I modified my topology to your...

That is very weird. I even added the secondary peerlink to the bond (as you have pictured in your topology but not configured) and I have not seen this error even once at any point in time. I'm starting to wonder about your version of Virtualbox. These UDP tunnels that are serving as the interconnects/links between the VMs are handled by Virtualbox. GNS3 simply passes the tunnel endpoint port-numbers to Virtualbox which builds the tunnels and passes the traffic; if traffic is not being passed correctly that would seem to be a Virtualbox issue. FYI: My versions are: Virtualbox : v5.0.10 r104061 Cumulus VX: Version 2.5.4 GNS3: v1.3.11. Of course if you're a Windows person there's the age-old "when in doubt, reboot" moniker but if it were me I would probably uninstall and re-install Virtualbox with a fresh download of whatever is current on their website.
Userlevel 5
Jim A wrote:

@Eric
lldpctl shows each leaf sees the other on swp1 and 2 (I modified my topology to your...

I've also collected every possible setting from the "Spine1" node in my config just in case you can find a setting that does not match what you have configured for your VM. You can view all the possible settings for a particular VM in virtualbox with "vboxmanage showvminfo " http://pastebin.com/cYzT5PYu

Reply