Passing tagged and untagged traffic on multiple bridges


I am using Cumulus Linux 2.5.7

I would like to pass VLAN tagged traffic and untagged traffic on multiple bridges on a Supermicro SSE-X3648. I am aware of a vlan-aware mode for bridges, but as I understand it, only one vlan-aware bridge can be present on the switch at any one time. I have tested vlan-aware bridging and am looking for a way to capture the vlan-aware behavior(passing all traffic regardless of tag) on multiple bridges up to 27 individual bridges.

I do see a flag that can be set in ifupdown2.conf:

# By default ifupdown2 only supports a single vlan filtering bridge# on the system. Set this flag to 1 to support multiple vlan
# filtering bridges
multiple_vlan_aware_bridge_support=1

Setting this flag does not appear to affect the behavior of the switch.

I came to the conclusion that I could not have multiple vlan-aware bridges and sought a less desirable solution. This involved specifying which individual vlans I would like to pass and adding the subinterfaces to separate bridges. For example, if I want to pass VLANs 1-10 between ports swp1 and swp2 I would write the following:

#cat /etc/network/interfaces.d/br1000
auto swp2
iface swp2
auto swp1
iface swp1
auto swp2.1
iface swp2.1
auto swp1.1
iface swp1.1
auto swp2.2
iface swp2.2
auto swp1.2
iface swp1.2
auto swp2.3
iface swp2.3
auto swp1.3
iface swp1.3
auto swp2.4
iface swp2.4
auto swp1.4
iface swp1.4
auto swp2.5
iface swp2.5
auto swp1.5
iface swp1.5
auto swp2.6
iface swp2.6
auto swp1.6
iface swp1.6
auto swp2.7
iface swp2.7
auto swp1.7
iface swp1.7
auto swp2.8
iface swp2.8
auto swp1.8
iface swp1.8
auto swp2.9
iface swp2.9
auto swp1.9
iface swp1.9
auto swp2.10
iface swp2.10
auto swp1.10
iface swp1.10
auto br1000
iface br1000
bridge-ports swp2 swp1
auto br1000-1
iface br1000-1
bridge-ports swp2.1 swp1.1
auto br1000-2
iface br1000-2
bridge-ports swp2.2 swp1.2
auto br1000-3
iface br1000-3
bridge-ports swp2.3 swp1.3
auto br1000-4
iface br1000-4
bridge-ports swp2.4 swp1.4
auto br1000-5
iface br1000-5
bridge-ports swp2.5 swp1.5
auto br1000-6
iface br1000-6
bridge-ports swp2.6 swp1.6
auto br1000-7
iface br1000-7
bridge-ports swp2.7 swp1.7
auto br1000-8
iface br1000-8
bridge-ports swp2.8 swp1.8
auto br1000-9
iface br1000-9
bridge-ports swp2.9 swp1.9
auto br1000-10
iface br1000-10
bridge-ports swp2.10 swp1.10

Thus creating 11 bridges. In order to determine the maximum VLAN range I can support in this fashion, I settled upon 200 after reading this document. I then made 27 connections in this fashion which results in 201 bridges per connection for a total of 5427 bridges. It took a while for the setup to complete and unfortunately the bridges do not pass traffic. I then continued to reduce the number of VLANs on each bridge until I found a number that will pass traffic over all the bridges. I came up with 16. According to my math, the total number of interfaces created in this fashion is thus:
(Vlans+1)*(ports per bridge +1)*(total number of bridges)=total interfaces

So in my example of 27 bridge with 16 VLANs, I created 1326 interfaces. It appears that I cannot add many more interfaces before switch behavior is compromised(traffic does not pass).

Now for the bad news, no error is thrown when exceeding this limit. If I check switchd.log I see:

2016-09-09T17:30:10.757871+00:00
cumulus switchd[24828]: hal_bcm.c:7294 CRIT Internal vlans exhausted
2016-09-09T17:30:10.757909+00:00 cumulus switchd[24828]: hal_bcm.c:6010 CRIT Cannot allocate bridge vlan for bridge id 12972

This seems like something that should not just be logged, but an error that should be handled. The switch shows the bridge as up and configured correctly which is misleading.

Phew, lot of text. I am finally at the point where I can ask my question. Since I know that a Broadcom Trident 2 can handle passing tagged and untagged traffic between multiple connections, can I use Cumulus Linux to control the chip in a way that creates multiple connections passing traffic regardless of VLAN tag? If not, is there a scheme that is less interface intensive? Am I right that the total interface count is the problem or is there some other limiting factor?

Thanks for reading.

14 replies

Userlevel 4

Supermicro SSE-X3648

If you are running a real switch feel free top open a case on https://support.cumulusnetworks.com

That being said

VLAN tagged traffic and untagged traffic on multiple bridges

You are providing a solution without providing the problem. We can't really help unless you tell us 'what' you are trying to achieve.

i.e. a problem would look like

"I need swp10 to have untagged VLAN 10 and tagged VLANs of 20,30,40"

Why do you need multiple bridges? The whole concept of the VLAN-aware bridge (also called the VLAN-filtering bridge) was to avoid the issue you are hitting (too many logical interfaces).
If you provide the problem you need we can help you with the config portion. I think you are getting caught up with the term bridges.
I apologize for not being clear.

I would like to pass VLAN tagged traffic and untagged traffic on multiple bridges on a Supermicro SSE-X3648.

What I meant was that I would like to have traffic ingress swp1 and egress swp2. At the same time I would like to have traffic ingress swp3 and egress swp4. At the same time I would like traffic to ingress swp5 and egress swp6, etc. I would like traffic to pass whether or not it is VLAN tagged. The way I have gone about this is to put the port pairs into bridges. Does this help clarify what I am trying to achieve?

Thank you for the response. I do indeed have a physical switch and a Cumulus license. Do you think that a support ticket is the way to go?
Userlevel 4
If this is high priority I would open a case. If you just want to get an answer 'eventually' this is the right place. We appreciate questions in the community because it helps everyone but high priority cases for paying customers take precedence.

Which VLANs can arrive on each pair? The answer cannot be 'all VLANs' unfortunately. You can't have all ports be separate for all VLANs (4096). What you are doing with the traditional bridge won't scale (hence the error).
I truly would like 'all VLANs' and I have seen first hand that this will not scale. Is there another way to configure the switch to capture the same behavior?
Userlevel 4
no, that is not possible. You can't have all ports have all VLANs.

What is the use case? It sounds like this is being used as a tap device?

What is the use case? It sounds like this is being used as a tap device?

It sure is!

My last hail Mary attempt was going to be to put all ports in one vlan-aware bridge and then use ACL ACCEPT/DROP rules to shape the traffic. Something like:

-A FORWARD -i swp1 -o swp2 -j ACCEPT
-A FORWARD -i swp3 -o swp4 -j ACCEPT
-A FORWARD -o swp2 -j DROP
-A FORWARD -o swp4 -j DROP

In order to create a 1to1 connection between swp1 and swp2, and a 1to1 connection between swp3 and swp4.

I have not had time to test traffic with this configuration but will report results. Any other recommendations or tap specific info?
Userlevel 4
We could do 2000 VLANs easily on all swp ports at the same time. We could a limited number of VLANs using old bridge driver on 'groupings' of ports. Or you can use SPAN/ERSPAN https://docs.cumulusnetworks.com/display/DOCS/Network+Troubleshooting#NetworkTroubleshooting-spanCon... which is really limited to just a few ports but captures everything.

If you don't explicitly allow a tag with your code above its not really going to do anything with it. Cumulus Linux is a switching and routing OS, not a tap device.
Could the cl-acltool handle 4096*54 rules?
Userlevel 4
Depends on the ASIC, not a CL limitation. Use cl-resource-query
Thanks will do!
Turns out that the rules I used above:

-A FORWARD -i swp1 -o swp2 -j ACCEPT
-A FORWARD -i swp3 -o swp4 -j ACCEPT
-A FORWARD -o swp2 -j DROP
-A FORWARD -o swp4 -j DROP

Do not result in swp1-->swp2 and swp3-->swp4. I added the following two rules:

-A FORWARD -o swp1 -j DROP
-A FORWARD -o swp3 -j DROP

Which the resulted in no traffic passing. I have a hunch that this may have to do with the order that the rules are applied. -o before -i if I remember correctly.
Userlevel 4
Bryan,

Are you still trying to accomplish the same goal as before? If so this is not going to work. I will go ahead and explain what these rules do:

If you have this->
-A FORWARD -i swp1 -o swp2 -j ACCEPT
-A FORWARD -i swp3 -o swp4 -j ACCEPT
-A FORWARD -o swp2 -j DROP
-A FORWARD -o swp4 -j DROP

This would say:
  1. allow any/all traffic (that is already allowed by the kernel) that came in swp1 that wants to go out swp2 to go
  2. allow any/all traffic (that is already allowed by the kernel) that came in swp3 that wants to go out swp4 to go
  3. anything that didn't hit the previous two rules and wants to go out swp2 is dropped
  4. anything that didn't hit the previous three rules and wants to go out swp4 is dropped
These rules' don't 'automatically' allow all VLANs (hence the 'that is already allowed by the kernel' statement). The only iptables rule that don't need a 'that is already allowed by the kernel' are the SPAN/ERSPAN which are specific to network ASICs like broadcom and mellanox so we added that functionality.

Does that make sense? If you email me at sean at cumulusnetworks.com I can get you a coupon to discount a bootcamp seat. Look at the classes here: https://cumulusnetworks.com/education/instructor-led-training/

This would say:

  1. allow any/all traffic (that is already allowed by the kernel) that came in swp1 that wants to go out swp2 to go
  1. allow any/all traffic (that is already allowed by the kernel) that came in swp3 that wants to go out swp4 to go
  1. anything that didn't hit the previous two rules and wants to go out swp2 is dropped
  1. anything that didn't hit the previous three rules and wants to go out swp4 is dropped

That is what I would expect as well. Unfortunately with these four ports in a bridge, untagged traffic did not pass through.

Thanks
Userlevel 4
Show me the config of /etc/network/interfaces. Is this on real hardware or VX? Did you use cl-acltool to 'push' the rules into hw.

Reply