Solved

Issues creating a two leaf, two spine topology using Cumulus VX and VMWare Fusion


I'm following the VMWare Fusion tutorial for creating a 2 leaf/2 spine topology and have run into an interesting issue, and would like some advice on how to troubleshoot this further.

I've set up all 4 VMs as per the tutorial, including both the Fusion VM configs and the VX interfaces/quagga configs.

The connectivity and routing works as expected with quagga running on any leaf/spine pair for example (leaf1 and (spine1 or spine2)) or (leaf2 and (spine1 or spine2)). However when I start quagga on a third VX - whether it's the second leaf or the second spine - the following occurs:

- Connectivity between all of the VX instances is lost
- The routing tables on all VX instances changes to remove all neighbors, and only includes routes to local interfaces.
- The VX instance command line responsiveness becomes very slow on all 3 routers
- Memory and CPU utilization on one of the spines jumps to 100% (as seen in the command 'top')
- Memory and CPU on the leaf is OK.
- If I have 2 leaves and 1 spine, that one spine maxes out CPU/memory
- If I have 1 leaf and 2 spines, one of the spines is fine and the other maxes out CPU/memory.
- I have repeated this several times and the spine that gets maxed out seems to be random

(BTW - I have included the fix for the /25 subnet and also decided to keep the OSPF unnumbered configs as explained in this prior posting).

When I stop quagga on any of the 3 routers, the routing tables of the remaining 2 routers running quagga are restored, connectivity is restored, and memory utilization drops back down to ~115MB (leaf) and ~55MB (spine).

If 3 or 4 routers run quagga long enough, the following message appears in /var/log/syslog showing that VX runs out of memory and maxes out the CPU.

2016-03-07T02:38:30.865179+00:00 cumulus kernel: [ 8340.312383] [18031] 105 18031 1044 24 0 0 0 ping
2016-03-07T02:38:30.865180+00:00 cumulus kernel: [ 8340.313398] [18032] 105 18032 1045 23 0 0 0 sh
2016-03-07T02:38:30.865181+00:00 cumulus kernel: [ 8340.314400] Out of memory: Kill process 1721 (dhclient) score 5 or sacrifice child
2016-03-07T02:38:30.865182+00:00 cumulus kernel: [ 8340.315462] Killed process 1721 (dhclient) total-vm:9864kB, anon-rss:2288kB, file-rss:12kB
2016-03-07T02:38:38.663902+00:00 cumulus jdoo[3354]: 'localhost' sysloadavg(15min) of 280.0 matches resource limit [sysloadavg(15min)>110.0]
2016-03-07T02:38:38.666660+00:00 cumulus jdoo[3354]: 'localhost' cpu system usage of 53.7% matches resource limit [cpu system usage>30.0%]
2016-03-07T02:38:38.826223+00:00 cumulus jdoo[3354]: 'chk_acl' file doesn't exist
2016-03-07T02:38:38.827227+00:00 cumulus jdoo[3354]: 'chk_acl' exec: /usr/share/cl-utilities/acl_check.sh
2016-03-07T02:39:10.061006+00:00 cumulus jdoo[3354]: 'localhost' sysloadavg(15min) of 284.0 matches resource limit [sysloadavg(15min)>110.0]
2016-03-07T02:39:10.061207+00:00 cumulus jdoo[3354]: 'localhost' mem usage of 97.7% matches resource limit [mem usage>90.0%]
2016-03-07T02:39:10.061511+00:00 cumulus jdoo[3354]: 'localhost' exec: /usr/cumulus/bin/cl-support
2016-03-07T02:39:10.107378+00:00 cumulus jdoo[3354]: 'localhost' cpu system usage of 82.3% matches resource limit [cpu system usage>30.0%]
2016-03-07T02:39:10.229628+00:00 cumulus jdoo[3354]: 'chk_acl' file doesn't exist
2016-03-07T02:39:10.230614+00:00 cumulus jdoo[3354]: 'chk_acl' exec: /usr/share/cl-utilities/acl_check.sh
2016-03-07T02:39:42.200616+00:00 cumulus jdoo[3354]: 'localhost' sysloadavg(15min) of 291.0 matches resource limit [sysloadavg(15min)>110.0]
2016-03-07T02:39:42.200940+00:00 cumulus jdoo[3354]: 'localhost' mem usage of 96.7% matches resource limit [mem usage>90.0%]
2016-03-07T02:39:42.200949+00:00 cumulus jdoo[3354]: 'localhost' mem usage of 96.7% matches resource limit [mem usage>90.0%]
2016-03-07T02:39:42.201501+00:00 cumulus jdoo[3354]: 'localhost' exec: /usr/cumulus/bin/cl-support
2016-03-07T02:39:42.201515+00:00 cumulus jdoo[3354]: 'localhost' cpu system usage of 80.4% matches resource limit [cpu system usage>30.0%]
2016-03-07T02:39:42.241607+00:00 cumulus jdoo[3354]: 'chk_acl' file doesn't exist
2016-03-07T02:39:42.241720+00:00 cumulus jdoo[3354]: 'chk_acl' exec: /usr/share/cl-utilities/acl_check.sh
2016-03-07T02:40:13.430971+00:00 cumulus jdoo[3354]: 'localhost' sysloadavg(15min) of 296.0 matches resource limit [sysloadavg(15min)>110.0]
2016-03-07T02:40:13.433808+00:00 cumulus jdoo[3354]: 'localhost' mem usage of 98.2% matches resource limit [mem usage>90.0%]
2016-03-07T02:40:13.433823+00:00 cumulus jdoo[3354]: 'localhost' mem usage of 98.2% matches resource limit [mem usage>90.0%]
2016-03-07T02:40:13.434216+00:00 cumulus jdoo[3354]: 'localhost' exec: /usr/cumulus/bin/cl-support
2016-03-07T02:40:13.434229+00:00 cumulus jdoo[3354]: Cannot fork a new process
2016-03-07T02:43:28.293999+00:00 cumulus sudo: cumulus : TTY=pts/0 ; PWD=/var/log ; USER=root ; COMMAND=/usr/sbin/service quagga stop
2016-03-07T02:43:28.414407+00:00 cumulus watchquagga[3075]: Terminating on signal


My configuration is:

VMWare Fusion 6.0.6 on Mac OSX 10.9.5
512MB RAM per VX instance (is this too low?)
1 processor per VX instance (is this too low?)


auto lo
iface lo inet loopback
address 10.2.1.1/32

auto eth0
iface eth0 inet dhcp

auto swp1
iface swp1
address 10.2.1.1/32

auto swp2
iface swp2
address 10.2.1.1/32

auto swp3
iface swp3
address 10.4.1.1/24



service integrated-vtysh-config

interface swp1
ip ospf network point-to-point

interface swp2
ip ospf network point-to-point

router-id 10.2.1.1

router ospf
ospf router-id 10.2.1.1
network 10.2.1.1/32 area 0.0.0.0
network 10.4.1.0/24 area 0.0.0.0



auto lo
iface lo inet loopback
address 10.2.1.2/32

auto eth0
iface eth0 inet dhcp

auto swp1
iface swp1
address 10.2.1.2/32

auto swp2
iface swp2
address 10.2.1.2/32

auto swp3
iface swp3
address 10.4.2.1/24



service integrated-vtysh-config

interface swp1
ip ospf network point-to-point

interface swp2
ip ospf network point-to-point

router-id 10.2.1.2

router ospf
ospf router-id 10.2.1.2
network 10.2.1.2/32 area 0.0.0.0
network 10.4.2.0/24 area 0.0.0.0



auto lo
iface lo inet loopback
address 10.2.1.3/32

auto eth0
iface eth0 inet dhcp

auto swp1
iface swp1
address 10.2.1.3/32

auto swp2
iface swp2
address 10.2.1.3/32

auto swp3
iface swp3



service integrated-vtysh-config

interface swp1
ip ospf network point-to-point

interface swp2
ip ospf network point-to-point

router-id 10.2.1.3

router ospf
ospf router-id 10.2.1.3
network 10.2.1.3/32 area 0.0.0.0




auto lo
iface lo inet loopback
address 10.2.1.4/32

# The primary network interface
auto eth0
iface eth0 inet dhcp

auto swp1
iface swp1
address 10.2.1.4/32

auto swp2
iface swp2
address 10.2.1.4/32

auto swp3
iface swp3



service integrated-vtysh-config

interface swp1
ip ospf network point-to-point

interface swp2
ip ospf network point-to-point

router-id 10.2.1.4

router ospf
ospf router-id 10.2.1.4
network 10.2.1.4/32 area 0.0.0.0


If quagga is started only on leaf1 and spine2, for example, behavior on these two VX instances is expected, these are the routes and top output :


cumulus# sh ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, A - Babel, T - Table,
> - selected route, * - FIB route

K>* 0.0.0.0/0 via 172.16.230.2, eth0
O 10.2.1.1/32 [110/10] is directly connected, lo, 00:50:18
C * 10.2.1.1/32 is directly connected, swp2
C * 10.2.1.1/32 is directly connected, swp1
C>* 10.2.1.1/32 is directly connected, lo
O>* 10.2.1.4/32 [110/20] via 10.2.1.4, swp1 onlink, 00:29:53
* via 10.2.1.4, swp2 onlink, 00:29:53
O 10.4.1.0/24 [110/10] is directly connected, swp3, 00:50:18
C>* 10.4.1.0/24 is directly connected, swp3
C>* 172.16.230.0/24 is directly connected, eth0



top - 05:38:50 up 5:22, 2 users, load average: 0.09, 0.06, 0.06
Tasks: 73 total, 1 running, 72 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 433568 total, 115972 used, 317596 free, 13636 buffers
KiB Swap: 0 total, 0 used, 0 free, 48480 cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
42618 quagga 15 -5 30924 1824 944 S 0.3 0.4 0:22.71 ospfd
1 root 20 0 10648 824 692 S 0.0 0.2 0:06.18 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:05.60 ksoftirqd/0
5 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u:0
6 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
7 root rt 0 0 0 0 S 0.0 0.0 0:00.25 watchdog/0



cumulus# sh ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, A - Babel, T - Table,
> - selected route, * - FIB route

K>* 0.0.0.0/0 via 172.16.230.2, eth0
O>* 10.2.1.1/32 [110/20] via 10.2.1.1, swp1 onlink, 00:30:10
* via 10.2.1.1, swp2 onlink, 00:30:10
O 10.2.1.4/32 [110/10] is directly connected, lo, 00:50:25
C * 10.2.1.4/32 is directly connected, swp2
C * 10.2.1.4/32 is directly connected, swp1
C>* 10.2.1.4/32 is directly connected, lo
O>* 10.4.1.0/24 [110/20] via 10.2.1.1, swp1 onlink, 00:30:10
* via 10.2.1.1, swp2 onlink, 00:30:10
C>* 172.16.230.0/24 is directly connected, eth0


If quagga is started on either leaf2 or spine1, the issue occurs, and the routes and top change to look like this :


cumulus# sh ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, A - Babel, T - Table,
> - selected route, * - FIB route

K>* 0.0.0.0/0 via 172.16.230.2, eth0
O 10.2.1.1/32 [110/10] is directly connected, lo, 00:55:45
C * 10.2.1.1/32 is directly connected, swp2
C * 10.2.1.1/32 is directly connected, swp1
C>* 10.2.1.1/32 is directly connected, lo
O 10.4.1.0/24 [110/10] is directly connected, swp3, 00:55:45
C>* 10.4.1.0/24 is directly connected, swp3
C>* 172.16.230.0/24 is directly connected, eth0



cumulus# sh ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, A - Babel, T - Table,
> - selected route, * - FIB route

K>* 0.0.0.0/0 via 172.16.230.2, eth0
O 10.2.1.4/32 [110/10] is directly connected, lo, 00:55:35
C * 10.2.1.4/32 is directly connected, swp2
C * 10.2.1.4/32 is directly connected, swp1
C>* 10.2.1.4/32 is directly connected, lo
C>* 172.16.230.0/24 is directly connected, eth0



top - 06:19:17 up 4:01, 1 user, load average: 233.66, 201.73, 89.17
Tasks: 2167 total, 2 running, 2165 sleeping, 0 stopped, 0 zombie
%Cpu(s): 12.0 us, 85.3 sy, 0.0 ni, 0.0 id, 0.2 wa, 0.0 hi, 2.5 si, 0.0 st
KiB Mem: 433568 total, 428704 used, 4864 free, 52 buffers
KiB Swap: 0 total, 0 used, 0 free, 1508 cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17776 quagga 15 -5 30924 900 36 S 11.0 0.2 0:04.94 ospfd
1 root 20 0 10648 132 0 S 5.8 0.0 0:48.93 init
21 root 20 0 0 0 0 R 5.2 0.0 1:27.74 kswapd0
43635 quagga 15 -5 4176 96 0 S 5.2 0.0 0:00.17 ping
43629 quagga 15 -5 4176 92 0 S 3.1 0.0 0:00.10 ping
40946 cumulus 20 0 23240 2232 200 R 2.1 0.5 0:00.14 top
43639 quagga 15 -5 4176 96 0 S 1.8 0.0 0:00.06 ping
43631 quagga 15 -5 4176 96 0 S 0.9 0.0 0:00.03 ping


And when quagga is stoped on leaf2/spine1, the routes are restored.

Thanks for any suggestions you can provide.

One other comment - actually a request for the next revision of the VX Fusion image - it'd be nice if /home/cumulus/.bashrc included this statement:

export VTYSH_PAGER=/bin/more


The default pager for vtysh on the Fusion OVA is cat. Adding the above line at the end of the bashrc file will change the default vtysh pager to something that will stop output from flying by faster than you can read it. This is very helpful in Fusion when using it with the tutorial, as the default terminal that starts up the VX VM doesn't have scroll bars. I found that it's likely best to just open up the Mac app Terminal and ssh into the DHCP IP address on eth0 - but until you can get that going - setting the default VTYSH pager to something other than cat would be useful.

2 replies

Userlevel 1
Hello Jonathan
I've got a suspect for your issue. Based on the output of your routing table it seems that all your Fusion interface are sharing the same vSwitch (you should not see OSPF equal cost multipath on that topology) . So basically all your interfaces seems on the same broadcast domain even if you are specifying "point-to-point" interface on your OSFP config. So as everything is unumbered you might end up with all routers trying to build a neighborship with everyone + all kind of weird thing. If you were not using Ip unumbered but a /31 instead on each point to point link you should not see that behavior. Unfortunately Fusion network settings are really poor and it's not easy to add multiple vswitch. You can use a tool like "UBER network fuser" (http://nickapedia.com/2012/01/10/breaking-new-ground-an-uber-tool-for-the-mac/) to create separate broadcast domain and making sure you emulate "point-to-point" type link.
jlouis
Thank you Jean-Louis, you were correct; I was able to use 4 separate "networks" in Fusion to create separate broadcast domains for the 4 pairs of point-to-point interfaces, kept IP unnumbered for those interfaces, and the example worked fine after that.

I used "UBER network fuser" (still works in OSX 10.9.5 / Fusion 6.0.6) to create the 4 additional Fusion "networks". Those networks appeared as 'Custom' networks in Fusion, and then I could assign the specific network adapters associated with the leaf/spine swp1/swp2 interface pairs into those 4 networks via the Fusion network adapter settings.

Reply