Cannot SSH into a VX VM running in QEMU


Userlevel 2
I've started a VX VM via GNS3 on my system; this is the command that was used by GNS3 to start the VM:

/usr/bin/qemu-system-x86_64 -name Cumulus-VX-sw1-1 -m 1024M -smp cpus=1 -enable-kvm -boot order=c -drive file=/home/gns3user/GNS3/projects/784f6727-93c6-4773-ad1c-676cfb11cd89/project-files/qemu/aeda2ad9-0446-4632-b47c-a60e2587f479/hda_disk.qcow2,if=ide,index=0,media=disk -serial telnet:0.0.0.0:2000,server,nowait -monitor tcp:127.0.0.1:57872,server,nowait -net user,vlan=0,net=192.168.0.0/24,hostfwd=tcp::1401-:22 -net none -device virtio-net-device,mac=00:00:ab:f4:79:00 -device virtio-net-device,mac=00:00:ab:f4:79:01 -device virtio-net-device,mac=00:00:ab:f4:79:02 -device virtio-net-device,mac=00:00:ab:f4:79:03 -nographic

When I try to SSH into the VM, it connects, but then hangs before presenting a login prompt. I see by the output of 'ss -tlnp' that it does have a listener at TCP/1401 ...

$ ss -tlnpState       Recv-Q Send-Q                                     Local Address:Port                                       Peer Address:Port 
LISTEN 0 128 *:111 *:*
LISTEN 0 1 *:2000 *:* users:(("qemu-system-x86",17771,14))
LISTEN 0 5 192.168.122.1:53 *:*
LISTEN 0 128 *:22 *:*
LISTEN 0 1 *:1401 *:* users:(("qemu-system-x86",17771,10))
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 1 127.0.0.1:52284 *:* users:(("qemu-system-x86",17771,7))
LISTEN 0 100 *:8000 *:* users:(("gns3server",6294,7))
LISTEN 0 128 :::111 :::*
LISTEN 0 128 :::22 :::*
LISTEN 0 100 ::1:25 :::*

Maybe it's the VM OS not booting properly, but without a console, I can't see that 😞

Anyone have any ideas on how to troubleshoot?

Thanks,
Will

15 replies

A couple of things, I can think of:
1) Use the options that we have specified in the User Guide. I had tested those, to make sure, that ssh works.
2) You could run the qemu cmd from cmdline, with the options that you have (remove the -nographic option). On tty/console. you should be able to see errors, if any.
Would be interested in knowing what you see by trying these.

Userlevel 2
Hi Naveed,

Thanks for the reply. I did just what you suggested in (2) above, got a console, and watched the boot. Only error I saw was a non-fatal failure of the "Installing default acl boot rules..." where it threw an error on line 29 that /usr/cumulus/bin/cl-acltool could not be found (bet you already are aware of this error though.) The boot process completed successfully, and I did get a login prompt on the console.

I logged in with user "cumulus", and did an "ip link show" and a "ip route show". I have eth0 and swp1-3; eth0 is in UP state, but I saw from a "ip addr show" that there is no IPv4 addr assigned to the interface. So how does one SSH to the box? I assume it's the part of the QEMU command line that reads "-net user,vlan=0,net=192.168.0.0/24,hostfwd=tcp::1401-:22" but how does that work? (What interface does this connect to on the VM?)

If I do an SSH session to TCP/1401 on localhost, I do see it connecting on IPv4, but it then hangs --
[root@gns3server01 ~]# ssh -vvvv -p 1401 cumulus@localhostOpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 56: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to localhost [::1] port 1401.
debug1: connect to address ::1 port 1401: Connection refused
debug1: Connecting to localhost [127.0.0.1] port 1401.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: SELinux support disabled
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1

Again, how does the connection to TCP/1401 get me into the VM??

Userlevel 2
Can anyone from the Cumulus staff please answer me question above? Would love to get this working!
Userlevel 5
Will,

What setup did you use in setting up Cumulus VX in GNS3? I would like to dig in more and help you out. Let me take a look on my side and we can get deeper.

Thanks,

Scott
Userlevel 2
Scott Suehle wrote:

Will,

What setup did you use in setting up Cumulus VX in GNS3? I would like to dig in more and ...

(I sent an email replying to this post, but see that it went to a "do-not-reply" email address instead of yours... So posting here...)

Hi Scott,
How do you mean “what setup”? How the settings are in the GNS3 GUI for the QEMU VM?

I followed the guide at: http://docs.cumulusnetworks.com/display/VX/Using+GNS3+with+QEMU+and+KVM+Virtual+Machines for the basic setup; the one thing I found is that for me at least, I could not use the older “virtio” driver, but had to use the newer “virtio-net-pci” driver for the NICs.

Here is an ASCII-art diagram of the (somewhat simplified) connection between my laptop running GNS3 client, and the server running GNS3 server... (expand to fit if necessary)
https://gist.githubusercontent.com/wd...
Userlevel 5
Will,

I am comparing your configuration to the demo configuration provided in the VX guide. Is there a reason you chose to setup the qemu line as you did? It looks like it is not parsing the port rules correctly and therefore 22 is not getting opened properly.

http://docs.cumulusnetworks.com/display/VX/Using+GNS3+with+QEMU+and+KVM+Virtual+Machines

If you have reason for setting your environment up as such please let me know and we can work on it a bit more to figure out the issue.

Userlevel 2
Here is a screenshot of my VX QEMU template:


Userlevel 2
Will Dennis wrote:

Here is a screenshot of my VX QEMU template:
[img]https://d1qy7qyune0vt1.cloudfront.net/cumulusne...

...and this is the command GNS3 is (now) using to instantiate a VM based on the above template:

2015-09-10 21:27:28 INFO qemu_vm.py:843 Starting QEMU with: /bin/qemu-system-x86_64 -name Cumulus-VX-sw1-1 -m 1024M -smp cpus=1 -enable-kvm -boot order=c -drive file=/home/gns3user/GNS3/projects/784f6727-93c6-4773-ad1c-676cfb11cd89/project-files/qemu/95b90747-b610-420f-9377-d4d6956d547e/hda_disk.qcow2,if=ide,index=0,media=disk -serial telnet:0.0.0.0:2000,server,nowait -monitor tcp:127.0.0.1:52303,server,nowait -net user,vlan=0,net=192.168.0.0/24,hostfwd=tcp::1401-:22 -net none -device virtio-net-pci,mac=00:00:ab:54:7e:00 -device virtio-net-pci,mac=00:00:ab:54:7e:01 -device virtio-net-pci,mac=00:00:ab:54:7e:02 -device virtio-net-pci,mac=00:00:ab:54:7e:03 -nographic
(notice it's a bit different from the original one I posted above, mainly the NIC device driver)
Userlevel 2
Will Dennis wrote:

Here is a screenshot of my VX QEMU template:
[img]https://d1qy7qyune0vt1.cloudfront.net/cumulusne...

The guide says this:
    In the QEMU VM configuration dialog, click the Network tab. Increase the number of Adapters to 4. Select the Type to be Legacy paravirtualized. Click Advanced Settings. In the options field, enter: -net user,vlan=0,net=192.168.0.0/24,hostfwd=tcp::1401-:22
If I set the Network tab Type dropbox to be "Legacy paravirtualized Network I/O" (QEMU NIC driver 'virtio') I get the following error when I try to start the VM:
qemu-system-x86_64: -device virtio,mac=00:00:ab:f4:79:00: 'virtio' is not a valid device model name
I found that if I switched the Type dropbox to be "Paravirtualized Network I/O" (QEMU NIC driver 'virtio-net-pci') then the VM starts fine.

I have set the Advanced Settings tab > Additional Settings > Options field to be the exact string specified in your doc. You can see from my GNS3 log file output above that it is using that exact string...
Userlevel 5
Will Dennis wrote:

Here is a screenshot of my VX QEMU template:
[img]https://d1qy7qyune0vt1.cloudfront.net/cumulusne...

Will, I have asked the engineer who put that section together too take a look. Please bear with us as we dig deeper into this.
Userlevel 2
Will Dennis wrote:

Here is a screenshot of my VX QEMU template:
[img]https://d1qy7qyune0vt1.cloudfront.net/cumulusne...

Great, thanks Scott! Let me know if you need any other info.
Will Dennis wrote:

Here is a screenshot of my VX QEMU template:
[img]https://d1qy7qyune0vt1.cloudfront.net/cumulusne...

Will:

Could you try the changes in Bold/Italic below:
  • In the QEMU VM configuration dialog, click the Network tab.
  • Increase the number of Adapters to 4.
  • -------------
  • Select the Checkbox, which states "Use the legacy networking mode".
  • Select the Type (from the drop down list) to be "Legacy Paravirtualized Network I/O".
  • -----------
  • Click Advanced Settings. In the options field, enter:
  • -net user,vlan=0,net=192.168.0.0/24,hostfwd=tcp::1401-:22

Userlevel 2
[quote=Will Dennis]Here is a screenshot of my VX QEMU template:
https://



Now here is the command line produced by GNS3:

2015-09-14 21:19:54 INFO qemu_vm.py:843 Starting QEMU with: /bin/qemu-system-x86_64 -name Cumulus-VX-sw1-1 -m 1024M -smp cpus=1 -enable-kvm -boot order=c -drive file=/home/gns3user/GNS3/projects/535c0f6c-bd3d-4b2a-ad29-c65d0cfefc9f/project-files/qemu/9bf6ef88-6432-4fd7-9fba-244d35606fd0/hda_disk.qcow2,if=ide,index=0,media=disk -serial telnet:0.0.0.0:2000,server,nowait -monitor tcp:127.0.0.1:57473,server,nowait -net user,vlan=0,net=192.168.0.0/24,hostfwd=tcp::1401-:22 -net none -net nic,vlan=0,macaddr=00:00:ab:6f:d0:00,model=virtio -net nic,vlan=1,macaddr=00:00:ab:6f:d0:01,model=virtio -net nic,vlan=2,macaddr=00:00:ab:6f:d0:02,model=virtio -net nic,vlan=3,macaddr=00:00:ab:6f:d0:03,model=virtio -nographic

...and now I do indeed get a successful login when I do a "ssh -p 1401 cumulus@localhost" on the server 🙂

I believe that the first time I tried your published config, that I did not check the "Use the legacy networking mode" checkbox (which is NOT specified in the doc), but did set the "Type:" field to "Legacy paravirtualized Network I/O" (which IS specified...) and hence got the " 'virtio' is not a valid device model name " error... I now see that checking that checkbox makes for a "-net ..." line instead of a "-device ..." line for each NIC. I am not that well-versed with QEMU networking; can you (briefly) explain what the difference is between using "Legacy" or not using it?

In any case, it is now working for me, and I thank all you Cumulus folk who assisted me in resolving this. Of course, someone should update the doc as to the need to check the checkbox for "Use the legacy networking mode" 🙂

Userlevel 3
Will Dennis wrote:

Here is a screenshot of my VX QEMU template:
[img]https://d1qy7qyune0vt1.cloudfront.net/cumulusne...

Ouch, sorry for the oversight in the docs, Will. I've updated the doc to reflect this.
Userlevel 2
Pointer to getting SSH access working (at least to one VX VM at the time of writing) when NOT using "legacy mode" -
https://community.cumulusnetworks.com/cumulus/topics/cannot-successfully-connect-vx-vms-in-gns3?topi...

Reply