Need help with setting Cumulus VX with vSphere

  • 8 January 2016
  • 27 replies
  • 6716 views

Hi,

I am new to Cumulus, I am following the below wiki for setting up leaf and spines on VMWare vSphere.

https://docs.cumulusnetworks.com/display/VX/Using+Cumulus+VX+with+VMware+vSphere+-+ESXi+5.5

I have followed all these steps, Issue which I am seeing now is I am not able to ping one another as mentioned in the Testing Connection section. We are using private network for all these Cumulus VM’s.

Thanks,
Charu

27 replies

after restarting the VMs with the proper capital Q quagga config files the outout of "show run" actually shows the entries for the ospf router:

router ospf
ospf router-id 10.2.1.1
network 10.2.1.1/32 area 0.0.0.0
network 10.4.1.0/24 area 0.0.0.0
!
ip forwarding
ipv6 forwarding
!
line vty
!
end

And now the ping to leaf2 actually works.

the routing table looks better now as well, since it does have routes for the leafs

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.74.2 0.0.0.0 UG 0 0 0 eth0
10.2.1.2 10.2.1.3 255.255.255.255 UGH 20 0 0 swp1
10.2.1.3 10.2.1.3 255.255.255.255 UGH 20 0 0 swp1
10.2.1.4 10.2.1.4 255.255.255.255 UGH 20 0 0 swp2
10.4.1.0 * 255.255.255.0 U 0 0 0 swp3
10.4.2.0 10.2.1.3 255.255.255.0 UG 20 0 0 swp1
192.168.74.0 * 255.255.255.0 U 0 0 0 eth0

thank you so much guys and please make sure to add all our findings to the tutorial.
as already mentioned, you could simply put 4 preconfigured VMs into on virtual appliance and let people download that. they would be up and running in 15 minutes instead of hacking at the commandline.
i did indeed create a quagga.conf (without the capital Q) because there was none in /etc/quagga.
i will rename them now and restart the VMs to check if that fixes some of the problems already

this is why i like open source software... why the hell would one want to have a capital letter in the name of a config file?
Userlevel 2
Great catch! I'll get that fix today. The file name should be Quagga.conf not quagga.conf.
Userlevel 4
@Jean-Louis, I would not edit the Quagga.conf file directly until you are 100% confident it won't break the config. Quagga has an industry standard CLI which included syntax checking. Just type sudo vtysh, then 'configure terminal' and you can add the configuration line by line. This will ensure the configuration is correct (it will error if your syntax is incorrect). Alternatively there is cl-ospf commands if you don't want to use the vtysh modal command line and you won't have to leave Bash.

You 'can' edit the Quagga.conf file directly, but this requires you do restart the quagga daemon (sudo service quagga restart). There is a smart quagga reload function as well that I won't get into on this thread.
Userlevel 1
@Jan: Your Quagga.conf seems to be empty
On the tutorial, it seems it make you create a file called quagga.conf with lowercase q.
Line vtysh create a Quagga.conf file with uppercase Q.
Make sure your config is on Quagga.conf with Q uppercase
Userlevel 2
Also, in your "show run" output you need to have a stanza for router ospf.
Userlevel 2
All of the OSPF neighbor links require "ip ospf network point-to-point" under the interface stanza in quagga for OSPF unnumbered to function correctly.
the output of "show run" is

root@cumulus-leaf1:~# vtysh

Hello, this is Quagga (version 0.99.23.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

cumulus-leaf1# show run
Building configuration...

Current configuration:
!
hostname zebra
log file /var/log/quagga/zebra.log
hostname ospfd
log file /var/log/quagga/ospfd.log
log timestamp precision 6
hostname bgpd
log file /var/log/quagga/bgpd.log
username cumulus nopassword
!
service integrated-vtysh-config
!
password cn321
enable password cn321
!
interface eth0
link-detect
!
interface lo
link-detect
!
interface swp1
link-detect
!
interface swp2
link-detect
!
interface swp3
link-detect
!
interface swp4
link-detect
!
interface swp5
link-detect
!
interface swp6
link-detect
!
interface swp7
link-detect
!
ip forwarding
ipv6 forwarding
!
line vty
!
end

and the routing table with the /32 setup does look like this:

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.74.2 0.0.0.0 UG 0 0 0 eth0
10.4.1.0 * 255.255.255.0 U 0 0 0 swp3
192.168.74.0 * 255.255.255.0 U 0 0 0 eth0

which is wrong, because it should have two path/uplinks to the spine layer, shouldn't it ?
Userlevel 1
@Jan. This is what you should see from leaf1:

Leaf1#
Leaf1# sh ip ospf neighbor


Neighbor ID Pri State Dead Time Address Interface RXmtL RqstL DBsmL
10.2.1.3 1 Full/DROther 36.712s 10.2.1.3 swp1:10.2.1.1 0 0 0
10.2.1.4 1 Full/DROther 39.762s 10.2.1.4 swp2:10.2.1.1 0 0 0


Leaf1#
will try that, thank you
Userlevel 1
@sean@David. Thanks and yes i realized you wanted to use unnumbered but it wasn't clear on your example.
@Jan. Could you change back all your PtP config with /32. I just built a Lab and everything is working as expected. But i had to use 2 documents to make it work. Once config is set could you do:
root@cumulus:~# sudo vtysh
cumulus# show run
Sorry, i'm an ex-CLI freak so i found it easier to troubleshoot..
Userlevel 2
Sorry for not jumping on this a little sooner, and I can add a little more color later today when I get into the office if needed. Here is the tl:dr. Jean-Louis is correct. The tutorial is utilizing OSPF Unnumbered which in turn utilizes a single /32 IP for all the OSPF neighbor point-to-point links. This /32 IP is also the same one the loopback is set to which saves on /30 IP addresses that would normally be needed for every link. Think of the savings! Check out this link for another example along with some color around OSPF Unnumbered and why it is useful. https://support.cumulusnetworks.com/hc/en-us/articles/202796476-OSPF-Unnumbered-Sample-Configurations
Userlevel 4
@Jean-Louis, if you were using /30s or /31s you would eat up a subnet per link. Imagine a leaf-spine network with 32 leaf switches and 6 spine switches. This would eat 192 subnets at minimum (6 spine switches * 32 leaf switches), just for the fabric. With OSPF unnumbered you only need 1 IP address per network switch. This was done on purpose in the tutorial. In the boot camp we actually shifted to OSPF numbered (normal /31s) so we wouldn't confuse students (learning linux, learning ospf unnumbered at same time). To read more about OSPF unnumbered look here: https://docs.cumulusnetworks.com/display/DOCS/Open+Shortest+Path+First+-+OSPF+-+Protocol#OpenShortestPathFirst-OSPF-Protocol-ConfigurationTip:UnnumberedInterfaces
thank you for looking into it, jean-louis
Userlevel 1
@Jan: Ok now I'm starting to understand the background behind that tutorial. They want to leverage OSPF unnumbered. Let me test it quickly and get back to you
thinking about it a little more, would there be the need for an ECMP setup anyways like TRILL or SPB to make both paths work, since now it is basically luch if on what interface the spine switch will forward the packet to. one works, one wont. Once the packet is on a spine switch there is no way for the spine switch to know if he reaches the target on swp1 or swp2 without putting a static route in there. since ospf will only propagate routes to subnets not towards actual endpoints.
Hey Scott, of course i can.
the hypervisor used is VMware Workstation 11, the interface setup is like this:



# The loopback network interface
auto lo
iface lo inet loopback
address 10.2.1.1/32
# The primary network interface
auto eth0
#iface eth0 inet dhcp
# Static setup
iface eth0 inet static
address 192.168.74.201
netmask 255.255.255.0
gateway 192.168.74.2
# Link from leaf1 to spine1
auto swp1
iface swp1
address 10.2.1.1/30
# Link from leaf1 to spine2
auto swp2
iface swp2
address 10.2.1.1/30
# Link from leaf1
auto swp3
iface swp3
address 10.4.1.1/24



# The loopback network interface
auto lo
iface lo inet loopback
address 10.2.1.2/32
# The primary network interface
auto eth0
#iface eth0 inet dhcp
# Static setup
iface eth0 inet static
address 192.168.74.202
netmask 255.255.255.0
gateway 192.168.74.2
auto swp1
iface swp1
address 10.2.1.2/30
auto swp2
iface swp2
address 10.2.1.2/30
auto swp3
iface swp3
address 10.4.2.1/24



# The loopback network interface
auto lo
iface lo inet loopback
address 10.2.1.3/32
# The primary network interface
auto eth0
#iface eth0 inet dhcp
# Static setup
iface eth0 inet static
address 192.168.74.203
netmask 255.255.255.0
gateway 192.168.74.2
auto swp1
iface swp1
address 10.2.1.3/30
auto swp2
iface swp2
address 10.2.1.3/30
auto swp3
iface swp3



# The loopback network interface
auto lo
iface lo inet loopback
address 10.2.1.4/32
# The primary network interface
auto eth0
#iface eth0 inet dhcp
# Static setup
iface eth0 inet static
address 192.168.74.204
netmask 255.255.255.0
gateway 192.168.74.2
auto swp1
iface swp1
address 10.2.1.4/30
auto swp2
iface swp2
address 10.2.1.4/30
auto swp3
iface swp3


To simulate direct links between the switches i created seperate VMnet (Host-Only) Network configurations (without DHCP Service & Host-Adapter in it). i used VMnet16, VMnet17, VMnet18 and VMnet19 as you can see in the picture below. I set that VMnet Config on both ends/interfaces of the lines, so VMnet 16 on swp1 of spine1 and swp1 of leaf1.



the Quagga.conf files are following (i adjusted them to incorporate the 30bit subnet discussed in this thread)


service integrated-vtysh-config
interface swp1
ip ospf network point-to-point
interface swp2
ip ospf network point-to-point
router-id 10.2.1.1
router ospf
ospf router-id 10.2.1.1
network 10.2.1.1/30 area 0.0.0.0
network 10.4.1.0/24 area 0.0.0.0



service integrated-vtysh-config
interface swp1
ip ospf network point-to-point
interface swp2
ip ospf network point-to-point
router-id 10.2.1.2
router ospf
ospf router-id 10.2.1.2
network 10.2.1.2/30 area 0.0.0.0
network 10.4.2.0/24 area 0.0.0.0



service integrated-vtysh-config
interface swp1
ip ospf network point-to-point
interface swp2
ip ospf network point-to-point
router-id 10.2.1.3
router ospf
ospf router-id 10.2.1.3
network 10.2.1.3/30 area 0.0.0.0



service integrated-vtysh-config
interface swp1
ip ospf network point-to-point
interface swp2
ip ospf network point-to-point
router-id 10.2.1.4
router ospf
ospf router-id 10.2.1.4
network 10.2.1.4/30 area 0.0.0.0



Now these are the outputs of the route command on the switches:

leaf1:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.74.2 0.0.0.0 UG 0 0 0 eth0
10.2.1.0 * 255.255.255.252 U 0 0 0 swp1
10.2.1.0 * 255.255.255.252 U 0 0 0 swp2
10.4.1.0 * 255.255.255.0 U 0 0 0 swp3
192.168.74.0 * 255.255.255.0 U 0 0 0 eth0

leaf2:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.74.2 0.0.0.0 UG 0 0 0 eth0
10.2.1.0 * 255.255.255.252 U 0 0 0 swp1
10.2.1.0 * 255.255.255.252 U 0 0 0 swp2
10.4.2.0 * 255.255.255.0 U 0 0 0 swp3
192.168.74.0 * 255.255.255.0 U 0 0 0 eth0

spine1:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.74.2 0.0.0.0 UG 0 0 0 eth0
10.2.1.0 * 255.255.255.252 U 0 0 0 swp1
10.2.1.0 * 255.255.255.252 U 0 0 0 swp2
192.168.74.0 * 255.255.255.0 U 0 0 0 eth0

spine2:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.74.2 0.0.0.0 UG 0 0 0 eth0
10.2.1.4 * 255.255.255.252 U 0 0 0 swp1
10.2.1.4 * 255.255.255.252 U 0 0 0 swp2
192.168.74.0 * 255.255.255.0 U 0 0 0 eth0

By my understanding a packet for leaf 2 (10.2.1.2) should traverse from leaf1 swp1 or swp2 (since both interface have the same metric) towards either spine1 or spine2. Now when the packet arrives at spine1 or spine2 those will lookup the next hop and either put it on swp1 (which means it is getting back to leaf1 instead of leaf2) or towards swp2 which would indeed forward it correctly.

thank you in advance
@Jan . Look at your routing table. You have 2 routes for the destination that you are tryi...hey Jean-Louis,
thank you for your reply. i just followed the tutorial instructions which tell you to bind the same IP on SWP1 and SWP2 on the leaf switches, see here

Userlevel 1
@Jan . Look at your routing table. You have 2 routes for the destination that you are trying to ping so you must have configured same IP on 2 interfaces. Look at my config above,I m using a totally different range to make it work. The loopback can stay the same
@scott. Hey Scott, can someone look at your tutorials and change the /32 on PtP interface? Otherwise it will not work
Thanks
Userlevel 5
Jan, Can you share more of your configuration? What hypervisor are you using? What is your interfaces setup?
guys, this looks like a good fix, but you really should fix that in all of your tutorials which still state a 32 bit subnet.


after changing the subnets for the switch ports to 30 bit the route table does look a bit richer, but there is still no ping possible between leaf1 and leaf2
Userlevel 1
Hello Charu
Based on the topology from the wiki, here is a quick fix:
Edit /etc/network/interfaces on each of the Spines/Leafs and replace the ip add with the following: (ifdown/ifup when you have finished editing)
#Leaf1

auto swp1
iface swp1
address 10.2.11.2/30

auto swp2
iface swp2
address 10.2.12.2/30
--------------------
#Leaf2

auto swp1
iface swp1
address 10.2.21.2/30

auto swp2
iface swp2
address 10.2.22.2/30
--------------------

#Spine1

auto swp1
iface swp1
address 10.2.11.1/30

auto swp2
iface swp2
address 10.2.21.1/30
--------------------

#Spine2

auto swp1
iface swp1
address 10.2.12.1/30

auto swp2
iface swp2
address 10.2.22.1/30
Thank for the information Jean.

@Scott, can you please help me out with proper steps.
Userlevel 1
huum interesting. I've looked at the wiki page you are referring and from what i can see there must be a typo. Both swp1 and swp2 are setup with /32 address so you can't reach the other side of the interface and bring your session up. swp1 and swp2 should have on that example a /30 netmask.
Thanks for the reply Scott and Jean. Promiscuous mode is already enabled.

Scott, please find the attached screenshot for 'ip addr show' command on Leaf-1. When I try to ping other leaf or spine i see "Network is unreachable" message.

My config files are same as mentioned in the wiki, I will attach them soon. I am currently not able to ssh to these Cumulus VMs. Any pointers would be helpful for SSH configuration.



Output after restarting networking service in leaf-1



Thanks,
Charu

Reply