ACL in ebtables not behaving as expected


Userlevel 1
So, just imagine a simple scenario with 2 hosts connected to 2 ports PORTA and PORTB.

And I have this in the policy file installed by cl-acltool :

[ebtables]
-A FORWARD -i $PORTA -j ACCEPT
-A FORWARD -o $PORTA -j ACCEPT
-A FORWARD -i $PORTB -j DROP

I should be able to ping between port A and port B without issue because as long as port A is either the ingress or egress port, there is an ACCEPT rule that will be hit. And from my knowledge of linux ebtables, that's how it'd work.

However AFAICT that's not what's happening on a real switch. It seems that the rules with a "-i" are both evaluated first, before the rule with a "-o", which is a massive semantic difference between linux ebtables which are in-order and what the real-hw seems to be executing.
(Of course that's just a theory, something else could be wrong and cause the same effect)

Even weirder the ARP and IP packets seem to be treated differently even though the rules have nothing ARP specific in there. The behavior for ARP is a bit erratic with some packets going through and some others that don't. The IP ones are consistently blocked.

6 replies

Userlevel 4
I would probably do it like this... and explicitly deny the traffic I don't want to flow.

Userlevel 1
Sure that indeed works (tested it even before posting) but this is N^2 rules to add, that grows pretty fast with the number of ports and those tables aren't unlimited. In the end I might just end up matching on the source/destination MACs of the "trusted" device because neither way actually works properly when you have a MLAG setup (when packets have crossed the ISL you don't have the port info of where they came from). Also, I always prefer whitelist approach than blacklist. If traffic I want to flow is blocked, I'll notice quickly. If OTOH traffic I want blocked is still going through, I might not notice until it's too late.

At this point I'm more interested to know why the above rules don't work.

I totally understand that the software linux tables are pretty much infinitely flexible and that it's impossible to map all of it to the HW ASIC to be done at line rate. However it's important that I can predict how my rules will behave and if the hw can't execute the rules the way I wrote them, it should just refuse to load them, just like when I try to load an unsupported match.

I tried looking at some of the broadcom stuff on http://github.com/Broadcom-Switch to see if I could catch some info about how things are actually done to see what's possible, but I'm not even sure that stuff is related to the Trident 2 HW.

Cheers,

Sylvain

Userlevel 4
Sylvain Munaut wrote:

Sure that indeed works (tested it even before posting) but this is N^2 rules to add, that grows p...

Just got out of a meeting with one of our L2 experts on this topic essentially these are just what you suspect. Limitations in HW Architecture. Our iptables/ebtables construct tries to layer the Linux implementation on top of the underlying HW but they are not always directly compatible. We have been working on trying to document more of the differences but it is a difficult process to produce documentation that makes sense while simultaneously not mentioning any of the underlying HW architecture (due to legal reasons) which necessitates the differences. You have correctly ascertained that "the rules with a "-i" are both evaluated first, before the rule with a "-o"". When talking with the L2 expert he suggested that you try adding an ingress wildcard match to turn the -o rule into a -i rule. I just updated our docs to cover the wildcard case "Currently only swp+ and bond+ are supported as wildcard names." So you might have to do a bit of interface re-naming to get this to work with the supported wildcards. [ebtables] -A FORWARD -i $PORTA -j ACCEPT -A FORWARD -i swp+ -o $PORTA -j ACCEPT -A FORWARD -i $PORTB -j DROP
Userlevel 1
Hi Eric,

Thanks for looking into this and being so responsive !

I tried the -i swp+ and indeed this seems to work.

And I think I also found the source of the ARP weirdness I was seeing.

When a packet is broadcasted, it will match any -o rule and get broadcasted to all ports (of the bridge), and not just to the ports that actually match the "-o" condition, so you could probably slip by packets by the ACL by abusing this.

If I had to guess, the packets isn't actually duplicated in hardware and so only one decision ACCEPT/DROP can be used.

I see you point for the documentation probably being hard to write and make sense without revealing the low level details. But at least throwing a warning when running cl-acltool would be useful to warn the user that maybe something won't behave as expected.

At least those limits seemed to input/output port matching and the ones on src mac/dst mac/protocol behaved much more predictably. Which I guess make sense since output port is quite a bit more "dynamic" (i.e. not known at ingress) while all the others can be done as ingress filters.

Cheers,

Sylvain
Userlevel 4
Sylvain Munaut wrote:

Hi Eric,

Thanks for looking into this and being so responsive !

I tried the -i swp+ and indeed ...

Per our L2 expert: So only ARP is getting through past the drop rule? Other packets are getting dropped properly? Are the counters on the rule incrementing? If the answer to these questions are yes, then I would think the ARP packets are likely locally (i.e., CPU) generated ARP packets. They will not hit this FORWARD chain rule. The FORWARD chain rules are only for packets transiting the switch.
Userlevel 1
I don't have access to the HW anymore (it was using your test lab) so I'll have to wait until I get my own to retest, but from memory :

No this wasn't limited to only ARP packets. It seemed to be any packet that was broadcasted to all ports (BUD packet) so :
- Anything with a dst mac ff:ff:ff:ff:ff:ff
- Anything with a dst mac that's not in the bridge known mac address table.

So for instance I was able to slip by one ICMP packet by :
- Injecting manually the dst mac in the source machine ARP table (so there is no ARP request)
- Waiting long enough so that the switch doesn't remember that dst mac

Cheers,

Sylvain

Reply