Load Balancing traffic on egress


I am using Cumulus Linux 2.5.7. I would like to load balance traffic across multiple ports as it leaves the switch. I do not have to ability to control or predict how the host machine will be configured.

In order to accomplish load balancing, I have created a bonded interface that enslaves destination ports and added this bond as a member of a bridge containing the source ports. Unfortunately, this configuration resembles a multicast not a LAG.

Here is what I implemented:

#cat /etc/network/interfaces
auto bond1iface bond1
bond-slaves swp3 swp2
bond-lacp-bypass-allow 1
bond-xmit-hash-policy layer3+4
bond-lacp-bypass-all-active 1

auto br0
iface br0
bridge-ports bond1 swp1

When passing traffic from swp1, both swp2 and swp3 get all packets. So for example, if I generate 10 packets with random MAC and IP sources, both swp2 and swp3 get all 10 packets. I need to implement a configuration where the traffic is load balanced across the ports without duplicate packets being received.

Please let me know if there is a way to configure bonded interfaces and bridges in a way that accomplishes this, or if I need to take a different approach.

Thanks.

9 replies

Userlevel 4
The mode of bonding that Cumulus is using here is called LACP (ieee 802.3ad). LACP operates such that packets are hashed to flow across a particular link in the bundle based on a hash of the packet's source and destination; more specifically, LACP selects the same link for each IP address and port combination (layer 3+4). "If I generate 10 packets with random MAC and IP sources, both swp2 and swp3 get all 10 packets." This behavior is not consistent with the operation of LACP which will only send traffic down an individual link in the bundle and should not duplicate packets/frames across any two links. Per the 802.3ad standard: This standard does not mandate any particular distribution algorithm(s); however, any distribution algorithm shall ensure that, when frames are received by a Frame Collector as specified in 5.2.3, the algorithm shall not cause: a) misordering of frames that are part of any given conversation, or b) duplication of frames In short, your configuration looks correct to achieve your goal. Could you tell me more about how it is being determined that traffic is being duplicated across both links? Maybe I can help walk through what you're seeing.
Userlevel 1
Hi, Bryan.
Looks like you use 'LACP bypass' feature - https://docs.cumulusnetworks.com/display/CL255/LACP+Bypass

This is not what you are looking for. I think, you need to remove the string 'bond-lacp-bypass-all-active 1' and 'bond-lacp-bypass-allow 1' from your config, because it makes your bond work like multicast device - receive on one port, transmit on all bond members.

Sergei.
Userlevel 4
Sergei Hanus wrote:

Hi, Bryan.
Looks like you use 'LACP bypass' feature - [url=https://docs.cumulusnetworks.com/displ...

Yes. I agree. This is likely the issue... You are defaulting to passive mode LACP in this configuration. Removing the line as Sergei suggests will enforce Active mode LACP which will not allow anything but the LACP bonded links to come up.
Thank you both for the replies.

I have been referring to this Technical Document which states that: bond-mode: Must be set to 802.3ad. Since I have no way of configuring LACP on the host machine, I attempted to use LACP-Bypass modes. After messing around a bit, I believe I have found a solution.

Just as Sergei states, the original interfaces file I posted results in a multicast.

If I remove 'bond-lacp-bypass-all-active 1' from my interfaces file, traffic only passes to one port in the bond.

Removing both lines as per Sergei's suggestion results in no traffic passing into the bond.

Interestingly enough, below is the interfaces file that I used to successfully load balance traffic:

#cat /etc/network/interfaces
auto bond1
iface bond1
bond-mode balance-rr
bond-slaves swp3 swp2
auto br0
iface br0
bridge-ports bond1 swp1

Not only did this balance traffic even though the documentation forbids bond modes other than 802.3ad, but this configuration only works for me when passing traffic with different IPV4 addressing. This leads me to believe that it is still hashing on layer3+4 even though round-robin was specified.

While the above configuration works for my purposes, is it possible to get some clarifying information regarding any of this?

Thanks again!
Userlevel 1
Speaking about balance-rr independent of Cumulus, it has its pros and cons.
If this mode is set, packets are transmitted in sequential order from the first available slave to the last.
Balance-rr will send packets across multiple interfaces, even packets that belong to the same TCP/IP connection. Probably, that's why you see traffic distribution even with one IP address - no actual hashing happens. The problem here is when utilizing multiple sending and multiple receiving links, packets are often received out of order, which result in segment retransmission. So, I guess, for small amount of traffic this may not be a big deal, but for a good load it might cause retransmissions.

As for statement about bonding modes support in Cumulus - I personally also thought, that 802.3ad is the only supported. Maybe, the word "supported" is the key here. Balance-rr may work, but has caveats, though unsupported officially.
Sergei Hanus wrote:

Speaking about balance-rr independent of Cumulus, it has its pros and cons.
If this mode is set, ...

There is no traffic distribution with one IP address. Traffic is distributed with traffic containing multiple IP addresses. There appears to be some hashing going on behind the scenes. I have tested traffic flows with random MAC address and also see no traffic distribution. So it appears to be hashing on layer3+4. I find this odd, since balance-rr has been specified.
Userlevel 4
Sergei Hanus wrote:

Speaking about balance-rr independent of Cumulus, it has its pros and cons.
If this mode is set, ...

What does cat /proc/net/bonding/ say? I am curious this worked at all unless you are on Cumulus VX...
Userlevel 4
Sergei Hanus wrote:

Speaking about balance-rr independent of Cumulus, it has its pros and cons.
If this mode is set, ...

I have heard rumors about other bonding modes functioning but just that they aren't tested or supported, knowing that, I've never bothered to play with them.
Sean, here is the output of the /proc file:

#cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: swp2
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:25:90:b2:16:d2
Slave queue ID: 0

Slave Interface: swp3
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:25:90:b2:16:d3
Slave queue ID: 0

Thanks for taking a look at this. I am not using Cumulus VX, I am using Cumulus Linux 2.5.7 on a Supermicro SSE box.

Reply