Running modern SaltStack (2018.3.0) on Cumulus

  • 4 April 2018
  • 9 replies
  • 436 views

  • Learner
  • 7 replies
Cumulus Documentation states that the ability to bind a minion to a specific port or IP is only present in recent versions, e.g. https://docs.saltstack.com/en/latest/ref/configuration/minion.html#source-interface-name. Upon running the bootstrap script, I'm able to get it installed with -X -d options, e.g.:

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 14.0px 'Liberation Mono'; color: #ffffff; background-color: #000000; background-color: rgba(0, 0, 0, 0.68)} span.s1 {font-variant-ligatures: no-common-ligatures}
sudo sh bootstrap-salt.sh -X -d git v2018.3.0

But the minion will fail to launch because python-tornado is too old (from initial searches). I haven't found a way around this yet, but in our environment, we need to bind to eth0/mgmt vrf, as we're running eBGP unnumbered, and we have issues communicating with the Salt Master. I'm just wondering:

- is there a known fix to bind the interface/IP on the salt-minion that comes with Cumulus?
- is there another repo I can install from with newer packages known to work?
- any luck with early-access repos (don't really want to enable for other reasons)

I'm open to any suggestions - it seems best to just run Ansible at this point? I really can't believe that Salt only recently added the binding options for the minion. Am I missing something obvious?

9 replies

Userlevel 2
I'm a little dense here, is the problem with the config or with the minion install? What version of Cumulus Linux are you using?

I was able to install the minion after adding the global debian repo to the apt sources.

-Pete
Userlevel 2
let me know how the testing goes. We'd love to have a KB about running the salt minion in a vrf. If you are binding to eth0 then it should only do route lookups in the mgmt vrf and operate as expected. "Should" being the keyword 🙂
I actually have this up and running in a similar fashion to what you are describing, the issue is that when you are running the salt minion on the VRF you have to add salt to the VRF services file and start/enable the service as salt-minion@mgmt.

here is a refrence doc that I used to get it running.

https://docs.cumulusnetworks.com/display/DOCS/Management+VRF

Other than that one little issue you don't have to do anything special on the salt/zeromq side to get it running properly.
v3.5.3 - I was missing the second repo from the https://support.cumulusnetworks.com/hc/en-us/articles/215248118-Configuring-SaltStack-on-Cumulus-Linux doc. That's definitely the path of least resistance, thanks! (not having it gets you 2014.something).
Okay, everything's going well. I still haven't looked into where this fits into the mgmt VRF doc/style (https://docs.cumulusnetworks.com/display/DOCS/Management+VRF ), but for now, I just included a line for eth0 in the minion configuration file.
let me know how the testing goes. We'd love to have a KB about running the salt minion in a vrf. ...I'd like to do some massive testing with a VX setup to determine exactly what the issues are. Disabling IPv6 completely solved the issue, even though I was using interface + IP(v4) from the minion configs. The issues were with the spines (at least in our configs).

Some notes for myself/others:
- hostname/dns/fqdn resolution
- disabling IPv6 on master vs. troubleshooting minion -> master comms
- referencing 'mgmt' vs. 'eth0' (works for ping) - also, config files, but is there a difference?

In this case, the mgmt VRFs are /30'ed off of an edge router (new setup), and bounce right back into the leaf (where the salt server is). I'll get a more realistic demo in VX (and go from there).

Edit: seeing this in the logs:
[WARNING ] Unable to connect to the Master using a specific source IP / port
[WARNING ] Consider upgrading to pyzmq >= 16.0.1 and libzmq >= 4.1.6

Also, noticed this issue in the Saltstack repo: https://github.com/saltstack/salt/issues/38744
let me know how the testing goes. We'd love to have a KB about running the salt minion in a vrf. ...https://docs.saltstack.com/en/latest/topics/troubleshooting/minion.html really helped, and running:
salt-minion -l debug
never has any issues - I can keep running commands and they never occasionally fail like when it's running as a daemon. I'll keep looking into it tomorrow, and try on VX when I have time.
Okay, assuming you're running a Cumulus Validated Design for eBGP unnumbered, the salt-minion will respond on the 'lo' address of 10.0.0.21 or 22 (at least for me). I can't get it to bind, which pointed me to this link for the minion

https://docs.saltstack.com/en/latest/ref/configuration/minion.html#source-interface-name

Warning
This option requires modern version of the underlying libraries used by the selected transport:
  • zeromq requires pyzmq >= 16.0.1 and libzmq >= 4.1.6
  • tcp requires tornado >= 4.5


On the Cumulus side:

cumulus@sw-spine1:mgmt-vrf:~$ sudo apt-cache policy libzmq3
libzmq3:
Installed: 4.0.5+dfsg-3
Candidate: 4.0.5+dfsg-3
Version table:
*** 4.0.5+dfsg-3 0
500 http://repo.saltstack.com/apt/debian/8/amd64/latest/ jessie/main amd64 Packages
100 /var/lib/dpkg/status
4.0.5+dfsg-2+deb8u1 0
500 http://ftp.us.debian.org/debian/ jessie/main amd64 Packages
cumulus@sw-spine1:mgmt-vrf:~$ sudo apt-cache policy python-zmq
python-zmq:
Installed: 14.4.0-1
Candidate: 14.4.0-1
Version table:
*** 14.4.0-1 0
500 http://ftp.us.debian.org/debian/ jessie/main amd64 Packages
100 /var/lib/dpkg/status
14.4.0-1 0
500 http://repo.saltstack.com/apt/debian/8/amd64/latest/ jessie/main amd64 Packages


Salt on Cumulus always seems to want to use the 'lo' IP for communication in this scenario, which has intermittent success. Ironically, it works 100% of the time when I run the minions manually in debug mode. I'll tcpdump that to see what the difference is.

Any ideas?
Okay, assuming you're running a Cumulus Validated Design for eBGP unnumbered, the salt-minion wil...To summarize:

  • the minions can't set source interface or IP because of older zmq/tornado packages
  • when the minion daemon is running manually (with/without debug mode), the minions use the FQDN IP as source, which is the eth0/mgmt-vrp in this case (which has zero issues)
  • for now, just going to run with -d, instead of using default init scripts (still not sure what environment change is doing this) - both are running as root, although I'm launching the daemon manually with sudo
[WARNING ] Unable to connect to the Master using a specific source IP / port
[WARNING ] Consider upgrading to pyzmq >= 16.0.1 and libzmq >= 4.1.6

Reply