ping between "svi"


have a couple of topologies set up wherein i use a bridge-group tied with an address (similar to the concept of svi in cisco-land). i've never been able to ping successfully between these addresses within the topology -- even though i've defined all of the bridge ports successfully, mac-addresses populate correctly, and arp even resolves.
when i run a tcpdump on the interfaces, i see the arp requests -- but no response. static arp entries on the device also don't help.

running cumulus vx 2.5.3 on vbox.

i can post additional configs if needed.

14 replies

Userlevel 5
Quinn, thanks for the question. Not sure why this is failing for you. If we could see your configuration file and also to see the output of ip addr show on both bridges and ip route show. This could help us see what is going on with the setup.
Userlevel 4
yeah what Scott said, post the /etc/network/interfaces for everything
so -- there's a bit of cruft in here -- as i tried several different things (originally, a test with vrr, then moving the ip address to a single swp).
topology is as follows: cn1, cn2, cn3 are all in a triangle (think of cn1, cn2 as distribution/collapsed core switches). cn3 is a host that is dual-homed to each cn1, cn2. off of cn3, i have cn4, acting as a "host" with an ip address in the subnet that i wish to ping. this is pretty common in "cisco-land" (where i come from) for simulating end-user connectivity without having to spin up a pc or so.

cn1 /etc/network/interfaces:
cumulus@cn1-dev$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet dhcp
# turn up the interfaces -- for what??!
auto swp1
iface swp1
auto swp2
iface swp2 inet static
address 172.16.100.1/24
# make the bridge over which we will walk
#auto br_100
#iface br_100
# bridge-ports swp1 swp2
# mstpctl-treeprio 8192
# bridge-stp on
# address 172.16.100.1/24
# Static setup
#iface eth0 inet static
# address 10.0.2.15
# netmask 255.255.255.0
# gateway 10.0.2.2
cn2 /etc/network/interfaces:
note: this device isn't factoring in to anything yet. i wanted just a single host to test with

cumulus@cn2-dev$ cat /etc/network/interfaces  # This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet dhcp
# Static setup
#iface eth0 inet static
# address 10.0.2.15
# netmask 255.255.255.0
# gateway 10.0.2.2
cn3 /etc/network/interfaces

cumulus@cn3-dev$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet dhcp
# lets get the interfaces turnt up
auto swp1
iface swp1
auto swp2
iface swp2
auto swp3
iface swp3
# lets bridge them all together
auto br_100
iface br_100
bridge-ports swp1 swp2 swp3
bridge-stp on
# Static setup
#iface eth0 inet static
# address 10.0.2.15
# netmask 255.255.255.0
# gateway 10.0.2.2
cn4 /etc/network/interfaces

cumulus@cn4-dev$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet dhcp
auto swp1
iface swp1 inet static
address 172.16.100.100/24
# gateway 172.16.100.1
# Static setup
#iface eth0 inet static
# address 10.0.2.15
# netmask 255.255.255.0
# gateway 10.0.2.2
cn1 ip addr show
cumulus@cn1-dev$ ip addr show
1: lo: mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:07:a5:d4 brd ff:ff:ff:ff:ff:ff
inet 192.168.128.122/24 brd 192.168.128.255 scope global eth0
inet6 fe80::a00:27ff:fe07:a5d4/64 scope link
valid_lft forever preferred_lft forever
3: swp1: mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
link/ether 08:00:27:96:bf:f9 brd ff:ff:ff:ff:ff:ff
4: swp2: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:3c:71:88 brd ff:ff:ff:ff:ff:ff
inet 172.16.100.1/24 scope global swp2
inet6 fe80::a00:27ff:fe3c:7188/64 scope link
valid_lft forever preferred_lft forever
5: swp3: mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 08:00:27:55:2b:7d brd ff:ff:ff:ff:ff:ff
6: swp4: mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 08:00:27:dc:53:8e brd ff:ff:ff:ff:ff:ff
7: swp5: mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 08:00:27:bc:b1:3c brd ff:ff:ff:ff:ff:ff
8: swp6: mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 08:00:27:0a:dd:ca brd ff:ff:ff:ff:ff:ff
9: swp7: mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 08:00:27:66:9d:72 brd ff:ff:ff:ff:ff:ff
cn2 ip addr show

cumulus@cn2-dev$ ip addr show
1: lo: mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:b7:3e:1d brd ff:ff:ff:ff:ff:ff
inet 192.168.128.133/24 brd 192.168.128.255 scope global eth0
inet6 fe80::a00:27ff:feb7:3e1d/64 scope link
valid_lft forever preferred_lft forever
3: swp1: mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 08:00:27:4d:10:11 brd ff:ff:ff:ff:ff:ff
4: swp2: mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 08:00:27:28:fd:33 brd ff:ff:ff:ff:ff:ff
5: swp3: mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 08:00:27:18:26:32 brd ff:ff:ff:ff:ff:ff
6: swp4: mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 08:00:27:14:cf:7d brd ff:ff:ff:ff:ff:ff
7: swp5: mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 08:00:27:e4:fc:ef brd ff:ff:ff:ff:ff:ff
8: swp6: mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 08:00:27:68:1f:fa brd ff:ff:ff:ff:ff:ff
9: swp7: mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 08:00:27:9a:07:87 brd ff:ff:ff:ff:ff:ff
cn3 ip addr show

cumulus@cn3-dev$ ip addr show
1: lo: mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:e5:35:fd brd ff:ff:ff:ff:ff:ff
inet 192.168.128.134/24 brd 192.168.128.255 scope global eth0
inet6 fe80::a00:27ff:fee5:35fd/64 scope link
valid_lft forever preferred_lft forever
3: swp1: mtu 1500 qdisc pfifo_fast master br_100 state UP qlen 1000
link/ether 08:00:27:38:1e:be brd ff:ff:ff:ff:ff:ff
4: swp2: mtu 1500 qdisc pfifo_fast master br_100 state DOWN qlen 1000
link/ether 08:00:27:8a:4f:19 brd ff:ff:ff:ff:ff:ff
5: swp3: mtu 1500 qdisc pfifo_fast master br_100 state UP qlen 1000
link/ether 08:00:27:0d:45:a7 brd ff:ff:ff:ff:ff:ff
6: swp4: mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 08:00:27:14:54:b5 brd ff:ff:ff:ff:ff:ff
7: swp5: mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 08:00:27:59:60:9d brd ff:ff:ff:ff:ff:ff
8: swp6: mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 08:00:27:28:26:25 brd ff:ff:ff:ff:ff:ff
9: swp7: mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 08:00:27:7c:ef:e2 brd ff:ff:ff:ff:ff:ff
11: br_100: mtu 1500 qdisc noqueue state UP
link/ether 08:00:27:0d:45:a7 brd ff:ff:ff:ff:ff:ff
inet6 fe80::a00:27ff:fe0d:45a7/64 scope link
valid_lft forever preferred_lft forever
cn4 ip addr show

cumulus@cn4-dev$ ip addr show
1: lo: mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:14:af:83 brd ff:ff:ff:ff:ff:ff
inet 192.168.128.135/24 brd 192.168.128.255 scope global eth0
inet6 fe80::a00:27ff:fe14:af83/64 scope link
valid_lft forever preferred_lft forever
3: swp1: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:dd:d8:7b brd ff:ff:ff:ff:ff:ff
inet 172.16.100.100/24 scope global swp1
inet6 fe80::a00:27ff:fedd:d87b/64 scope link
valid_lft forever preferred_lft forever
4: swp2: mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 08:00:27:d2:5f:4d brd ff:ff:ff:ff:ff:ff
5: swp3: mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 08:00:27:c9:31:05 brd ff:ff:ff:ff:ff:ff
6: swp4: mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 08:00:27:50:56:03 brd ff:ff:ff:ff:ff:ff
7: swp5: mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 08:00:27:3b:60:47 brd ff:ff:ff:ff:ff:ff
8: swp6: mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 08:00:27:2b:2b:e3 brd ff:ff:ff:ff:ff:ff
9: swp7: mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 08:00:27:99:0d:52 brd ff:ff:ff:ff:ff:ff

the only routing occuring is through the eth0 interface. i've not created a separate namespace in this "dev" environment -- as i didn't feel it worth the effort. all devices should be in the same subnet -- so all that should be required is arp. as i stated -- i can see the arp entries correct on cn1 and cn4.

cn1
cumulus@cn1-dev$ arp -n  Address                  HWtype  HWaddress           Flags Mask            Iface
10.128.128.128 ether 00:18:0a:15:e3:40 C eth0
172.16.100.100 ether 08:00:27:dd:d8:7b CM swp2
192.168.128.1 ether 00:22:90:32:f3:e1 C eth0
192.168.128.102 ether d8:eb:97:b3:b7:d6 C eth0
cn4
cumulus@cn4-dev$ arp -n
Address HWtype HWaddress Flags Mask Iface
192.168.128.102 ether d8:eb:97:b3:b7:d6 C eth0
172.16.100.1 ether 08:00:27:07:a5:d4 CM swp1
192.168.128.1 ether 00:22:90:32:f3:e1 C eth0
in the interest of full disclosure -- as stated earlier -- i have a very strong cisco background and have worked in the partner community for about a decade. the things that i'm doing are generally what we do in "cisco-land" and may or may not be the "best way" to do things in cumulus. if there is a flaw in my methodology, please let me know as i'm interested in learning this software. however -- just based on what i'm seeing -- everything should just "work".

thanks!

q.
Userlevel 1
Hi Quinn,

Just a hunch, do you have promiscuous mode enabled in the vbox adapter properties for the interfaces on cn3? I am not certain about virtualbox, but in ESX you need to set the vSwitch interfaces to promiscuous mode for this kind of forwarding.

I'm guessing the swps on cn3 may not be passed a frame with a destination MAC (:a5:d4) which is foreign to cn3 (owned by cn1) unless promiscuous mode is enabled. If cn3 doesn't see it, it doesn't forward it.

Promiscuous mode may not help directly, if not it should at least open up your tcpdump visibility somewhat.
i don't have that enabled. i wondered about it -- but i guess my thought train was the following:
  • mac-addresses are being populated in the table (as evidenced by arp -n on the hosts in question).
  • even a static arp entry placed on both hosts *should* (in theory) negate the requirement for arp to occur in the first place, as the arp request would be negated.
  • ping between the hosts should be a unicast based on the source/dest ip address and forwarded by the device using source/dest mac addys.
i'll enable promiscuous mode on the adapters in question. i'll see if that makes a change.

thanks!

q.
so i enabled 'promiscuous mode' on all internal vbox adapters (just to be safe and get it all done at once). ping now works.

that being said -- i guess i'm missing why thats required -- especially since the control plane (which is actually doing the flooding of the arps) looks to be working -- unless there is something unique about why this is different in cumulus?

q.
Userlevel 4
https://en.m.wikipedia.org/wiki/Promiscuous_mode Without promiscuous mode it can still see unicast, multicast and broadcast (arp), it just won't process anything else (icmp to another node). This isn't anything that is related to Cumulus Linux but rather a hypervisor thing .
Userlevel 1
I suspect the problem began after ARP was already complete. With the MAC resolved, cn4 transmitted a ICMP echo request with L3 dest 172.16.100.1 and L2 dest :a5:d4. The L3 dest was fine, but the L2 dest was foreign.

Virtual bridges, especially in type 2 hypervisors like Virtualbox, usually connect to L2 endpoints (vs. forwarders) so the default is to drop frames with foreign L2 destination addresses. It's better for performance and security. When you have an L2 forwarder as a guest, it needs to see frames with L2 destinations that are not its own MACs. That's where promiscuous mode comes in. It's a configuration issue typical to virtual bridges and virtual switches, not specific to Cumulus VX.
sean --
thanks for that. i'm aware of what the promiscuous mode 'means' -- just didn't know its applicability in this situation. i've not set up 'virtual switches' (most of my nfv stuff has been on purpose-built virtual machines of cisco operating systems, wherein i've not tried to do l2transport trickery, etc).

i guess this would expand to include the following situation (i believe i have this spun up in my other environment) -- wherein i've a swp interface carrying a vlan (tagged, untagged, doesn't matter) tied to an svi in that same vlan. because the mac-addresses are not directly connected neighbors -- promiscuous mode must be required for the swp to pass the frames to the virtual-bridge, yes?

q.
Userlevel 4
Also I highly recommend using the add on tool netshow if you are coming from "Cisco land". It's not for configuring but it is a lot easier than using the ip add show commands for troubleshooting: https://support.cumulusnetworks.com/hc/en-us/articles/204075083-Installing-and-Using-the-cl-show-netshow-Troubleshooting-Tool Also the bridge you are using is in traditional mode you might want to check out the vlan-aware mode:https://support.cumulusnetworks.com/hc/en-us/articles/204909397-Comparing-Traditional-Bridge-Mode-to-VLAN-aware-Bridge-Mode Also the inet static are not required on CL, it's smart enough to know what you are up to 🙂 less typing! Glad you figured out your problem!
Userlevel 4
because the mac-addresses are not directly connected neighbors -- promiscuous mode must be required for the swp to pass the frames to the virtual-bridge, yes? Exactly! That's my take on it at least. The L2 Mac won't be processed unless it's it's own unicast, broadcast FF:FF:FF:FF:FF:FF or multicast 01-00-5E-00-00-00 to 01-00-5E-7F-FF-FF. Again like Steven said VMware, VBox and KVM could all do this slightly differently.
sean --

thanks again. i've installed the netshow package in my actual "test" environment. the 'dev' environment shows were just something quick and dirty i spun up that i could debug with rather than mucking with saving configs around (because i will forget to reapply something).

funny as it sounds -- the 'traditional mode' config is actually a bit more natural to me -- given that most of the equipment i support (asr9k, etc) running ios-xr (which runs on the qnx microkernel) doesn't have a concept of a vlan -- just a tagged interface that you can use to apply policy to (ethernet flow point -- efp). this is in contrast to (what i gathered at least) from the vlan-aware mode -- in which you have a concept of a vlan-database (similar to that which exists within the catalyst series switches). obviously -- for more "switching" events -- the vlan-aware config would be much more beneficial.

would it be possible to provide some caveat within the documentation -- or some sticky -- in regards to the requirement of promiscuous mode within the vx-mode? obviously -- this is something unique to the hypervisor space -- but after digging through a lot of docs -- i couldn't find anything conclusive.

thanks!

q.
Userlevel 4
Yeah that's a very reasonable request. We will get it added ASAP. Be aware that with traditional mode you are limited to around 200 vlans (tags). With the vlan-aware we can do 2000. Both of these limits are "soft limits" but something to keep in mind.
i can see where that would be an issue in traditional networks -- but vxlan fabric using evpn makes that a little less important i would imagine -- as at that point -- you're concerned more with ipv4 route space.

thanks for the update to the docs! it is greatly appreciated (although -- in my scripts to deploy future environments -- i'm just going to automatically set promisc mode on all adapters).

thanks again for the prompt support! very appreciated.

q.

Reply