vrf and mgmt-vrf packages from Cumulus Networks


In reading through your tutorial related to VRF's (http://schd.ws/hosted_files/ossna2017/fe/vrf-tutorial-oss.pdf) I've come to a point where I am having troubles getting services on a vrf and note that the article mentions vrf & mgmt-vrf packages from Cumulus & includes a link (https://github.com/CumulusNetworks/vrf) however there are not configuration files such that we could just pull the git repo and run 'configure', 'make' and then 'make install'.
Does that exist elsewhere/can you point me to other documentation that might get me past this point?
I am specifically to a point of switch services to@mgmt that fail due to missing files/directories (which are all auto-generated in Cumulus) thus I believe I'm missing something that is in the referenced package. This is on an Ubuntu host: 16.04 on 4.14 kernel with updated iproute2 and the ifupdown2 package that the above referenced tutorial suggested.

15 replies

hi Troy: At the moment that code base is just text files. You can run 'make install' to install files, 'make rpm' to create an rpm package to install or 'dpkg-buildpackage -uc -us' to build a deb package to install. I will add a README with that information as well as a 'make deb' target for the debian package.
thanks for the quick response David, I'll keep an eye on git for the update & give it a try

David: We were able to get several things working but I'm still stumbling on a few items. I can't seem to get rsyslog nor snmp@ working.
On the rsyslog I've added a directive specifying "Device=mgmt" but I'm getting an omfwd error.
On snmp I'm getting the following logs whenever I poll the device externally (we can run an snmpwalk & get responses locally but nothing off the box): "snmpd[3673]: send response: Failure in sendto"
I added a forwarding rule and got rsyslog sending via the vrf but prior to that it seems it was unable to connect. Does this sound familiar/should I need to add the forwarding rule to the FIB?
Have you seen this snmp behavior before/have any suggestions?

Thanks in advance.

Troy: Looks like my last response was eaten by goblins, so I'll try again.

rsyslog needs to be version 8.24 or higher. Ubuntu 16.04 has an older version.

net-snmp I need to check. At one point we needed a patched net-snmp to avoid it adding IP_PKTINFO with an ifindex of 0 which essentially removes the vrf binding done by the vrf command. That patch has been reverted and I tested net-snmp in Cumulus Linux yesterday and it worked fine with mgmt vrf.
I loaded rsyslog 8.32 (v8stable repo latest) and now I'm getting:
rsyslogd: No UDP socket could successfully be initialized, some functionality may be disabled. [v8.32.0]
rsyslogd: create UDP socket bound to device failed: Operation not permitted [v8.32.0]

This is with the following in the rsyslog configuration:
$template GRAYLOGRFC5424,"<%PRI%>%PROTOCOL-VERSION% %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n"
action( type="omfwd" Target="10.100.21.30" Device="mgmt" Port="1514" Protocol="udp" template="GRAYLOGRFC5424" )

Hmmm.... sounds like rsyslog is not running as root. That is the only way to get EPERM denied. Do this:
strace -o /tmp/rsyslogd.trace -fF -tt -T /usr/sbin/rsyslogd -n

Take a look at the trace file see which operation specifically is failing.
nice catch - it was running as syslog; running as root it seems to be just fine. Thanks
now I just need to get the snmp part - is there a specific release you would recommend?
snmpd release is not clear. I can see that the Ubuntu 16.04 version definitely has the sendmsg / IP_PKTINFO bug and that is why you get the sendto failures. I'll need to find some time to compare upstream code to the Cumulus version and see what change has it working. Perhaps early next week.
David - any news on an snmpd release to go after or is there some way that you're aware of that I can configure around the IP_PKTINFO bug?

Not good news for you, unfortunately. Someone is working on a solution for net-snmp to take upstream to properly handle net-snmp with VRFs.

I did submit a kernel patch that keeps net-snmp from overriding the VRF binding. I need to look at getting that applied to stable kernels. If you roll your own kernel, the commit is 1cbec07649ec ("net: Only honor ifindex in IP_PKTINFO if non-0")

The short of it is there is no solution in the near term that is going to be in the Ubuntu or Debian releases.
Bummer but thanks for the update, I'll work around it for now & move on and plan to circle back later.
Troy: The kernel patch is making its way to the stable trees (4.4, 4.9, 4.14, etc). Should find its way into the distributions in the next few months. At that point you will be able to run net-snmp over mgmt-vrf.
Troy:

The kernel patch I submitted upstream has been backported to 4.4 and 4.9. If you upgrade to kernel that is based on the kernel.org LTS kernels versions 4.4.126 and 4.9.92 or higher should have the fix that allows snmpd to run in a VRF context.
David,
We're looking at rolling out a few more systems and I'm looking at Debian this time instead of Ubuntu and wanted to check in & see if you had any insight or gotchas there like you provided for me with Ubuntu. Any issues with stretch (9.4) or should I stick with one of the newer Ubuntu bases?
Thanks
Troy: Any release with a kernel version 4.10 and higher and iproute2 4.10 and higher. I believe Debian stretch defaults are 4.9; if you go with stretch get the backports for both packages.

Reply