Vagrant fails to SSH to Device


Apologies for the wall of text below.

I am running Ubuntu server 16.04.1 LTS, with Vagrant version 1.8.7, and Virtualbox 5.036. After running through the quickstart on the EVPN demo https://github.com/CumulusNetworks/cldemo-evpn I am not able to get past powering on the first set of VM's. If anyone has any suggestions I would appreciate it. Thanks!

===The initial problem===

root@vagrant:/home/mike/cldemo-evpn# vagrant reload oob-mgmt-server==> oob-mgmt-server: Attempting graceful shutdown of VM...
oob-mgmt-server: Guest communication could not be established! This is usually because
oob-mgmt-server: SSH is not running, the authentication information was changed,
oob-mgmt-server: or some other networking issue. Vagrant will force halt, if
oob-mgmt-server: capable.
==> oob-mgmt-server: Forcing shutdown of VM...
==> oob-mgmt-server: Checking if box 'yk0/ubuntu-xenial' is up to date...
==> oob-mgmt-server: Clearing any previously set forwarded ports...
==> oob-mgmt-server: Clearing any previously set network interfaces...
==> oob-mgmt-server: Preparing network interfaces based on configuration...
oob-mgmt-server: Adapter 1: nat
oob-mgmt-server: Adapter 2: intnet
oob-mgmt-server: Adapter 3: intnet
==> oob-mgmt-server: Forwarding ports...
oob-mgmt-server: 22 (guest) => 2222 (host) (adapter 1)
==> oob-mgmt-server: Running 'pre-boot' VM customizations...
==> oob-mgmt-server: Booting VM...
==> oob-mgmt-server: Waiting for machine to boot. This may take a few minutes...
oob-mgmt-server: SSH address: 127.0.0.1:2222
oob-mgmt-server: SSH username: vagrant
oob-mgmt-server: SSH auth method: private key
Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.

If you look above, you should be able to see the error(s) that
Vagrant had when attempting to connect to the machine. These errors
are usually good hints as to what may be wrong.

If you're using a custom box, make sure that networking is properly
working and you're able to connect to the machine. It is a common
problem that networking isn't setup properly in these boxes.
Verify that authentication configurations are also setup properly,
as well.

If the box appears to be booting properly, you may want to increase
the timeout ("config.vm.boot_timeout") value.

===Additional Output===

root@vagrant:/home/mike/cldemo-evpn# vagrant statusCurrent machine states:

oob-mgmt-server running (virtualbox)
oob-mgmt-switch aborted (virtualbox)
exit02 not created (virtualbox)
exit01 not created (virtualbox)
spine02 not created (virtualbox)
spine01 not created (virtualbox)
leaf04 not created (virtualbox)
leaf02 not created (virtualbox)
leaf03 not created (virtualbox)
leaf01 not created (virtualbox)
edge01 not created (virtualbox)
server01 not created (virtualbox)
server03 not created (virtualbox)
server02 not created (virtualbox)
server04 not created (virtualbox)
internet not created (virtualbox)

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run 'vagrant status NAME'.
root@vagrant:/home/mike/cldemo-evpn# vagrant port oob-mgmt-server
The forwarded ports for the machine are listed below. Please note that
these values may differ from values configured in the Vagrantfile if the
provider supports automatic port collision detection and resolution.

22 (guest) => 2222 (host)
root@vagrant:/home/mike/cldemo-evpn# vagrant ssh-config
Host oob-mgmt-server
HostName 127.0.0.1
User vagrant
Port 2222
UserKnownHostsFile /dev/null
StrictHostKeyChecking no
PasswordAuthentication no
IdentityFile /root/.vagrant.d/insecure_private_key
IdentitiesOnly yes
LogLevel FATAL

The provider for this Vagrant-managed machine is reporting that it
is not yet ready for SSH. Depending on your provider this can carry
different meanings. Make sure your machine is created and running and
try again. Additionally, check the output of 'vagrant status' to verify
that the machine is in the state that you expect. If you continue to
get this error message, please view the documentation for the provider
you're using.

12 replies

Userlevel 4
What computer / OS are you running vagrant from?
Ubuntu server 16.04.1 LTS
Userlevel 5
That version of virtualbox is a bit older but I don't think that's the issue...

Perhaps try changing the "reboot" line ( https://github.com/CumulusNetworks/cldemo-evpn/blob/master/Vagrantfile#L53 ) to read:
nohup bash -c 'sleep 10; shutdown now -r "Rebooting to Remap Interfaces"' &  
Running a "vagrant destroy -f" and then starting the simulation again.

If that doesn't work you may want to install a newer version of virtualbox (something like 5.1.18+).
Replacing line 53 in the Vagrantfile did not work for me. I upgraded Virtualbox to 5.1.18, but still getting the same SSH error. What base OS are these demos built on? Perhaps I could try another Linux distribution?
Userlevel 4
I tested this on Ubuntu 16.04 but with KVM. I tested Virtual box is OSX. In the mean time since you are already running Ubuntu can you try these instructions for KVM that Eric wrote https://getsatisfaction.cumulusnetworks.com/cumulus/topics/setting-up-an-ubuntu-16-04-server-for-simulation-with-libvirt-kvm This isn't nested virtualization is it? Ubuntu on top of something else?
It is nested. I am running Ubuntu in Virtual Box on a Windows 7 PC. I have hardware virtualization enabled for the Ubuntu VM, but definitely possible this could be causing a problem.

I will run through the instructions Eric wrote and let you know how that goes.
Userlevel 4
You could run the Vagtantfile on Windows with Virtual box as well. I haven't had the best luck nesting although a few customers have gotten it to work.
Yeah, probably not worth beating my head against the wall when really all I want to do is demo EVPN. I went ahead and took your advice and installed Vagrant on Windows. The VM's are booting and I can SSH to them as well. Looks like this put me back on track. Thanks for your help Sean and Eric! In case anyone else needs instructions for Windows, I found these two articles:

https://github.com/CumulusNetworks/cldemo-vagrant-onwindows
https://www.sitepoint.com/getting-started-vagrant-windows/
Userlevel 5
Virtualbox does not support nested virtualization that sounds like the issue.
Have customers had success nesting within ESXi?
Userlevel 4
I know Eric helped one of our customers with ESXi and got nested to work . It is possible to get working please record any pains or breakthroughs you have here so we know.
Definitely. I do not think I am going to pursue Virtual Box nesting, but after I kick the tires on my desktop I would like to nest on ESXi in a larger environment. I will be sure to share my experience w/ the community.

Reply