Need help with setting Cumulus VX with vSphere


Hi,

I am new to Cumulus, I am following the below wiki for setting up leaf and spines on VMWare vSphere.

https://docs.cumulusnetworks.com/display/VX/Using+Cumulus+VX+with+VMware+vSphere+-+ESXi+5.5

I have followed all these steps, Issue which I am seeing now is I am not able to ping one another as mentioned in the Testing Connection section. We are using private network for all these Cumulus VM’s.

Thanks,
Charu

27 replies

Userlevel 5
Can you share you config files? Also what is the result of running ip addr show? Are all of the connections showing as UP?
Userlevel 1
Hello.
Make sure the port-group in your vswitch is setup in Promiscuous Port.
To enable promiscuous mode on a particular port group, go to ESXi Server and choose Configuration > Networking > Properties. Select the port group and click Edit. On the Security tab, check the box to enable Promiscuous Mode
Jlouis
Thanks for the reply Scott and Jean. Promiscuous mode is already enabled.

Scott, please find the attached screenshot for 'ip addr show' command on Leaf-1. When I try to ping other leaf or spine i see "Network is unreachable" message.

My config files are same as mentioned in the wiki, I will attach them soon. I am currently not able to ssh to these Cumulus VMs. Any pointers would be helpful for SSH configuration.



Output after restarting networking service in leaf-1



Thanks,
Charu
Userlevel 1
huum interesting. I've looked at the wiki page you are referring and from what i can see there must be a typo. Both swp1 and swp2 are setup with /32 address so you can't reach the other side of the interface and bring your session up. swp1 and swp2 should have on that example a /30 netmask.
Thank for the information Jean.

@Scott, can you please help me out with proper steps.
Userlevel 1
Hello Charu
Based on the topology from the wiki, here is a quick fix:
Edit /etc/network/interfaces on each of the Spines/Leafs and replace the ip add with the following: (ifdown/ifup when you have finished editing)
#Leaf1

auto swp1
iface swp1
address 10.2.11.2/30

auto swp2
iface swp2
address 10.2.12.2/30
--------------------
#Leaf2

auto swp1
iface swp1
address 10.2.21.2/30

auto swp2
iface swp2
address 10.2.22.2/30
--------------------

#Spine1

auto swp1
iface swp1
address 10.2.11.1/30

auto swp2
iface swp2
address 10.2.21.1/30
--------------------

#Spine2

auto swp1
iface swp1
address 10.2.12.1/30

auto swp2
iface swp2
address 10.2.22.1/30
guys, this looks like a good fix, but you really should fix that in all of your tutorials which still state a 32 bit subnet.


after changing the subnets for the switch ports to 30 bit the route table does look a bit richer, but there is still no ping possible between leaf1 and leaf2
Userlevel 5
Jan, Can you share more of your configuration? What hypervisor are you using? What is your interfaces setup?
Userlevel 1
@Jan . Look at your routing table. You have 2 routes for the destination that you are trying to ping so you must have configured same IP on 2 interfaces. Look at my config above,I m using a totally different range to make it work. The loopback can stay the same
@scott. Hey Scott, can someone look at your tutorials and change the /32 on PtP interface? Otherwise it will not work
Thanks
Jean-Louis Auzepy wrote:

@Jan . Look at your routing table. You have 2 routes for the destination that you are tryi...

hey Jean-Louis,
thank you for your reply. i just followed the tutorial instructions which tell you to bind the same IP on SWP1 and SWP2 on the leaf switches, see here


Hey Scott, of course i can.
the hypervisor used is VMware Workstation 11, the interface setup is like this:



# The loopback network interface
auto lo
iface lo inet loopback
address 10.2.1.1/32
# The primary network interface
auto eth0
#iface eth0 inet dhcp
# Static setup
iface eth0 inet static
address 192.168.74.201
netmask 255.255.255.0
gateway 192.168.74.2
# Link from leaf1 to spine1
auto swp1
iface swp1
address 10.2.1.1/30
# Link from leaf1 to spine2
auto swp2
iface swp2
address 10.2.1.1/30
# Link from leaf1
auto swp3
iface swp3
address 10.4.1.1/24



# The loopback network interface
auto lo
iface lo inet loopback
address 10.2.1.2/32
# The primary network interface
auto eth0
#iface eth0 inet dhcp
# Static setup
iface eth0 inet static
address 192.168.74.202
netmask 255.255.255.0
gateway 192.168.74.2
auto swp1
iface swp1
address 10.2.1.2/30
auto swp2
iface swp2
address 10.2.1.2/30
auto swp3
iface swp3
address 10.4.2.1/24



# The loopback network interface
auto lo
iface lo inet loopback
address 10.2.1.3/32
# The primary network interface
auto eth0
#iface eth0 inet dhcp
# Static setup
iface eth0 inet static
address 192.168.74.203
netmask 255.255.255.0
gateway 192.168.74.2
auto swp1
iface swp1
address 10.2.1.3/30
auto swp2
iface swp2
address 10.2.1.3/30
auto swp3
iface swp3



# The loopback network interface
auto lo
iface lo inet loopback
address 10.2.1.4/32
# The primary network interface
auto eth0
#iface eth0 inet dhcp
# Static setup
iface eth0 inet static
address 192.168.74.204
netmask 255.255.255.0
gateway 192.168.74.2
auto swp1
iface swp1
address 10.2.1.4/30
auto swp2
iface swp2
address 10.2.1.4/30
auto swp3
iface swp3


To simulate direct links between the switches i created seperate VMnet (Host-Only) Network configurations (without DHCP Service & Host-Adapter in it). i used VMnet16, VMnet17, VMnet18 and VMnet19 as you can see in the picture below. I set that VMnet Config on both ends/interfaces of the lines, so VMnet 16 on swp1 of spine1 and swp1 of leaf1.



the Quagga.conf files are following (i adjusted them to incorporate the 30bit subnet discussed in this thread)


service integrated-vtysh-config
interface swp1
ip ospf network point-to-point
interface swp2
ip ospf network point-to-point
router-id 10.2.1.1
router ospf
ospf router-id 10.2.1.1
network 10.2.1.1/30 area 0.0.0.0
network 10.4.1.0/24 area 0.0.0.0



service integrated-vtysh-config
interface swp1
ip ospf network point-to-point
interface swp2
ip ospf network point-to-point
router-id 10.2.1.2
router ospf
ospf router-id 10.2.1.2
network 10.2.1.2/30 area 0.0.0.0
network 10.4.2.0/24 area 0.0.0.0



service integrated-vtysh-config
interface swp1
ip ospf network point-to-point
interface swp2
ip ospf network point-to-point
router-id 10.2.1.3
router ospf
ospf router-id 10.2.1.3
network 10.2.1.3/30 area 0.0.0.0



service integrated-vtysh-config
interface swp1
ip ospf network point-to-point
interface swp2
ip ospf network point-to-point
router-id 10.2.1.4
router ospf
ospf router-id 10.2.1.4
network 10.2.1.4/30 area 0.0.0.0



Now these are the outputs of the route command on the switches:

leaf1:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.74.2 0.0.0.0 UG 0 0 0 eth0
10.2.1.0 * 255.255.255.252 U 0 0 0 swp1
10.2.1.0 * 255.255.255.252 U 0 0 0 swp2
10.4.1.0 * 255.255.255.0 U 0 0 0 swp3
192.168.74.0 * 255.255.255.0 U 0 0 0 eth0

leaf2:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.74.2 0.0.0.0 UG 0 0 0 eth0
10.2.1.0 * 255.255.255.252 U 0 0 0 swp1
10.2.1.0 * 255.255.255.252 U 0 0 0 swp2
10.4.2.0 * 255.255.255.0 U 0 0 0 swp3
192.168.74.0 * 255.255.255.0 U 0 0 0 eth0

spine1:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.74.2 0.0.0.0 UG 0 0 0 eth0
10.2.1.0 * 255.255.255.252 U 0 0 0 swp1
10.2.1.0 * 255.255.255.252 U 0 0 0 swp2
192.168.74.0 * 255.255.255.0 U 0 0 0 eth0

spine2:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.74.2 0.0.0.0 UG 0 0 0 eth0
10.2.1.4 * 255.255.255.252 U 0 0 0 swp1
10.2.1.4 * 255.255.255.252 U 0 0 0 swp2
192.168.74.0 * 255.255.255.0 U 0 0 0 eth0

By my understanding a packet for leaf 2 (10.2.1.2) should traverse from leaf1 swp1 or swp2 (since both interface have the same metric) towards either spine1 or spine2. Now when the packet arrives at spine1 or spine2 those will lookup the next hop and either put it on swp1 (which means it is getting back to leaf1 instead of leaf2) or towards swp2 which would indeed forward it correctly.

thank you in advance

thinking about it a little more, would there be the need for an ECMP setup anyways like TRILL or SPB to make both paths work, since now it is basically luch if on what interface the spine switch will forward the packet to. one works, one wont. Once the packet is on a spine switch there is no way for the spine switch to know if he reaches the target on swp1 or swp2 without putting a static route in there. since ospf will only propagate routes to subnets not towards actual endpoints.
Userlevel 1
@Jan: Ok now I'm starting to understand the background behind that tutorial. They want to leverage OSPF unnumbered. Let me test it quickly and get back to you
thank you for looking into it, jean-louis
Userlevel 4
@Jean-Louis, if you were using /30s or /31s you would eat up a subnet per link. Imagine a leaf-spine network with 32 leaf switches and 6 spine switches. This would eat 192 subnets at minimum (6 spine switches * 32 leaf switches), just for the fabric. With OSPF unnumbered you only need 1 IP address per network switch. This was done on purpose in the tutorial. In the boot camp we actually shifted to OSPF numbered (normal /31s) so we wouldn't confuse students (learning linux, learning ospf unnumbered at same time). To read more about OSPF unnumbered look here: https://docs.cumulusnetworks.com/display/DOCS/Open+Shortest+Path+First+-+OSPF+-+Protocol#OpenShortestPathFirst-OSPF-Protocol-ConfigurationTip:UnnumberedInterfaces
Userlevel 2
Sorry for not jumping on this a little sooner, and I can add a little more color later today when I get into the office if needed. Here is the tl:dr. Jean-Louis is correct. The tutorial is utilizing OSPF Unnumbered which in turn utilizes a single /32 IP for all the OSPF neighbor point-to-point links. This /32 IP is also the same one the loopback is set to which saves on /30 IP addresses that would normally be needed for every link. Think of the savings! Check out this link for another example along with some color around OSPF Unnumbered and why it is useful. https://support.cumulusnetworks.com/hc/en-us/articles/202796476-OSPF-Unnumbered-Sample-Configurations
Userlevel 1
@sean@David. Thanks and yes i realized you wanted to use unnumbered but it wasn't clear on your example.
@Jan. Could you change back all your PtP config with /32. I just built a Lab and everything is working as expected. But i had to use 2 documents to make it work. Once config is set could you do:
root@cumulus:~# sudo vtysh
cumulus# show run
Sorry, i'm an ex-CLI freak so i found it easier to troubleshoot..
will try that, thank you
Userlevel 1
@Jan. This is what you should see from leaf1:

Leaf1#
Leaf1# sh ip ospf neighbor


Neighbor ID Pri State Dead Time Address Interface RXmtL RqstL DBsmL
10.2.1.3 1 Full/DROther 36.712s 10.2.1.3 swp1:10.2.1.1 0 0 0
10.2.1.4 1 Full/DROther 39.762s 10.2.1.4 swp2:10.2.1.1 0 0 0


Leaf1#
the output of "show run" is

root@cumulus-leaf1:~# vtysh

Hello, this is Quagga (version 0.99.23.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

cumulus-leaf1# show run
Building configuration...

Current configuration:
!
hostname zebra
log file /var/log/quagga/zebra.log
hostname ospfd
log file /var/log/quagga/ospfd.log
log timestamp precision 6
hostname bgpd
log file /var/log/quagga/bgpd.log
username cumulus nopassword
!
service integrated-vtysh-config
!
password cn321
enable password cn321
!
interface eth0
link-detect
!
interface lo
link-detect
!
interface swp1
link-detect
!
interface swp2
link-detect
!
interface swp3
link-detect
!
interface swp4
link-detect
!
interface swp5
link-detect
!
interface swp6
link-detect
!
interface swp7
link-detect
!
ip forwarding
ipv6 forwarding
!
line vty
!
end

and the routing table with the /32 setup does look like this:

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.74.2 0.0.0.0 UG 0 0 0 eth0
10.4.1.0 * 255.255.255.0 U 0 0 0 swp3
192.168.74.0 * 255.255.255.0 U 0 0 0 eth0

which is wrong, because it should have two path/uplinks to the spine layer, shouldn't it ?

Userlevel 2
All of the OSPF neighbor links require "ip ospf network point-to-point" under the interface stanza in quagga for OSPF unnumbered to function correctly.
Userlevel 2
Also, in your "show run" output you need to have a stanza for router ospf.
Userlevel 1
@Jan: Your Quagga.conf seems to be empty
On the tutorial, it seems it make you create a file called quagga.conf with lowercase q.
Line vtysh create a Quagga.conf file with uppercase Q.
Make sure your config is on Quagga.conf with Q uppercase
Userlevel 2
Great catch! I'll get that fix today. The file name should be Quagga.conf not quagga.conf.
Userlevel 4
@Jean-Louis, I would not edit the Quagga.conf file directly until you are 100% confident it won't break the config. Quagga has an industry standard CLI which included syntax checking. Just type sudo vtysh, then 'configure terminal' and you can add the configuration line by line. This will ensure the configuration is correct (it will error if your syntax is incorrect). Alternatively there is cl-ospf commands if you don't want to use the vtysh modal command line and you won't have to leave Bash.

You 'can' edit the Quagga.conf file directly, but this requires you do restart the quagga daemon (sudo service quagga restart). There is a smart quagga reload function as well that I won't get into on this thread.

Reply