Solved

static route leaking between vrf's to access public internet

  • 20 November 2018
  • 1 reply
  • 142 views

Hello cumulus community,

we are using Cumulus VX 3.7.1 to experiment with Vagrant 2.2.1 and Libvirt 4.6.0.

We set up a leaf1. Within that leaf we want to separate networks using VRF (Virtual Routing and Forwarding).

What we did:
  • within the leaf1 we set up a vrf "pxe" having "swp1" enslaved to separate networks
  • we enabled static route leaking and borrowed a route from "default" vrf to access public internet that is accessible from eth0
Expected behavior:
We can access internet at OSI layer 3 (ICMP) AND layer 4 (TCP/IP).

Experienced behavior:
  1. ICMP is working.
  2. There is no response from internet access at layer 4. See the detailed setup below.
We think that "NAT" is required to have a route back from eth0 into the "pxe" vrf and swp1.

Question:
What is the missing part so we can have a response being passed back to swp1?

Thank you very much,
ize0j10

------------------------------------------------------------------------------------------------------------------

The complete setup for context

The relevant Vagrant box setup is:

code:
[...]
device.vm.hostname = "leaf1"
device.vm.box = "CumulusCommunity/cumulus-vx"
device.vm.box_version = "3.7.1"
[...]



We set up the VRF using:

code:
# enable static route leaking vrf_route_leak_enable = TRUE
vagrant@leaf1:~$ sudo vi /etc/cumulus/switchd.conf
vagrant@leaf1:~$ sudo systemctl restart switchd

# create vrf "pxe" having swp1 as member
vagrant@leaf1:~$ net add vrf pxe
vagrant@leaf1:~$ net add interface swp1 vrf pxe
vagrant@leaf1:~$ net commit


This leads to:
code:
vagrant@leaf1:~$ ip a s
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 52:54:00:51:bc:bd brd ff:ff:ff:ff:ff:ff
inet 192.168.121.41/24 brd 192.168.121.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::5054:ff:fe51:bcbd/64 scope link
valid_lft forever preferred_lft forever
3: swp1: mtu 1500 qdisc pfifo_fast master pxe state UP group default qlen 1000
link/ether 44:38:39:00:00:11 brd ff:ff:ff:ff:ff:ff
inet6 fe80::4638:39ff:fe00:11/64 scope link
valid_lft forever preferred_lft forever
[...]
6: pxe: mtu 65536 qdisc pfifo_fast state UP group default qlen 1000
link/ether de:c7:3f:43:d0:1c brd ff:ff:ff:ff:ff:ff



With this setup there is no internet access (as expected):
code:
vagrant@leaf1:~$ ping -I swp1 heise.de
connect: Network is unreachable


Now we added a static route to access internet via "default" vrf:

code:
net add routing route 0.0.0.0/0 192.168.121.1 vrf pxe nexthop-vrf default


Now ping (ICMP) is successful, but curl (TCP/IP) is not:

code:
vagrant@leaf1:~$ ping -I swp1 heise.de
ping: Warning: source address might be selected on device other than swp1.
PING heise.de (193.99.144.80) from 192.168.121.41 swp1: 56(84) bytes of data.

vagrant@leaf1:~$ sudo curl --verbose --interface swp1 --location heise.de/favicon.ico
* Hostname was NOT found in DNS cache
* Trying 193.99.144.80...


What we tried:

We figured out using tcpdump that the response from the curl to heise.de (193.99.144.85) is received at leaf1 (192.168.121.41) eth0 but not passed to swp1.

code:
vagrant@leaf1:~$ sudo tcpdump -i eth0 -n src 193.99.144.85                                    
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
07:55:44.019633 IP 193.99.144.85.443 > 192.168.121.41.59128: Flags [S.], seq 3951821356, ack 757416282, win 3912, options [mss 1304,nop,wscale 2,nop,nop,TS val 807284746 ecr 210730,sackOK,eol], length 0
07:55:44.050801 IP 193.99.144.85.443 > 192.168.121.41.59128: Flags [.], ack 518, win 1107, options [nop,nop,TS val 807284777 ecr 210760], length 0
[...]
icon

Best answer by ize0j10 20 November 2018, 13:14

Please close the question since it contains erroneous information. The setup is mixed up. I will open another more precise question.

Thanks
View original

This topic has been closed for comments

1 reply

Please close the question since it contains erroneous information. The setup is mixed up. I will open another more precise question.

Thanks