Cannot successfully connect VX VMs in GNS3


Userlevel 2
Now for my next problem concerning VX in GNS3 😛

I have two VX VMs in a topology. Followed the instructions in http://docs.cumulusnetworks.com/display/VX/Using+GNS3+with+QEMU+and+KVM+Virtual+Machines and I configured swp1-3 on both sides, then "ifup"d them. They do show as up and running in the OS:
cumulus@vx-1401$ ip link sh  1: lo:  mtu 16436 qdisc noqueue state UNKNOWN mode DEFAULT 
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
link/ether 00:00:ab:f0:1e:00 brd ff:ff:ff:ff:ff:ff
3: swp1: mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
link/ether 00:00:ab:f0:1e:01 brd ff:ff:ff:ff:ff:ff
4: swp2: mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
link/ether 00:00:ab:f0:1e:02 brd ff:ff:ff:ff:ff:ff
5: swp3: mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
link/ether 00:00:ab:f0:1e:03 brd ff:ff:ff:ff:ff:ff
So, then I tried to connect them with a link, but I get an error and the QEMU VMs stop:



The whole text from the GNS3 console is:

QEMU process has stopped, return code: 1  
Start QEMU with /usr/bin/qemu-system-x86_64 -name Cumulus-VX-port1402 -m 256M -smp cpus=1 -enable-kvm -boot order=c -drive file=/home/gns3user/GNS3/projects/e1caa8ec-06a2-44e3-aca0-7af8bb02272b/project-files/qemu/f35cd3ce-fcb9-4fe1-84ec-631edf37b48a/hda_disk.qcow2,if=ide,index=0,media=disk -serial telnet:0.0.0.0:2001,server,nowait -monitor tcp:127.0.0.1:54942,server,nowait -net user,vlan=0,net=192.168.0.0/24,hostfwd=tcp::1402-:22 -net none -net nic,vlan=0,macaddr=00:00:ab:b4:8a:00,model=virtio -net nic,vlan=1,macaddr=00:00:ab:b4:8a:01,model=virtio -net udp,vlan=1,name=gns3-1,sport=10001,dport=10000,daddr=192.168.1.165 -net nic,vlan=2,macaddr=00:00:ab:b4:8a:02,model=virtio -net nic,vlan=3,macaddr=00:00:ab:b4:8a:03,model=virtio -nographic
Execution log:
qemu-system-x86_64: -net udp,vlan=1,name=gns3-1,sport=10001,dport=10000,daddr=192.168.1.165: Invalid parameter 'udp'
QEMU process has stopped, return code: 1
Start QEMU with /usr/bin/qemu-system-x86_64 -name Cumulus-VX-port1401 -m 256M -smp cpus=1 -enable-kvm -boot order=c -drive file=/home/gns3user/GNS3/projects/e1caa8ec-06a2-44e3-aca0-7af8bb02272b/project-files/qemu/2bb07a7b-4d6d-4379-97ce-54c138d5f01e/hda_disk.qcow2,if=ide,index=0,media=disk -serial telnet:0.0.0.0:2000,server,nowait -monitor tcp:127.0.0.1:57483,server,nowait -net user,vlan=0,net=192.168.0.0/24,hostfwd=tcp::1401-:22 -net none -net nic,vlan=0,macaddr=00:00:ab:f0:1e:00,model=virtio -net nic,vlan=1,macaddr=00:00:ab:f0:1e:01,model=virtio -net udp,vlan=1,name=gns3-1,sport=10000,dport=10001,daddr=192.168.1.165 -net nic,vlan=2,macaddr=00:00:ab:f0:1e:02,model=virtio -net nic,vlan=3,macaddr=00:00:ab:f0:1e:03,model=virtio -nographic
Execution log:
qemu-system-x86_64: -net udp,vlan=1,name=gns3-1,sport=10000,dport=10001,daddr=192.168.1.165: Invalid parameter 'udp'

Anyone have any ideas about this, or should I post this on the GNS3 forums? (not sure who to turn to with this one...)

FYI, I am running QEMU 2.0.0 (latest version from yum repos) on CentOS 7.1

15 replies

Userlevel 2
In further testing, if I make the QEMU VM templates have "new style" network adapters (i.e., "Use the legacy networking mode" is UNchecked, and Type set to "virtio-net-pci") then when I form a link between the instantiated VMs from this template, the link does work (goes green and stays up) and I see that the relevant QEMU config line on the VMs is:
-netdev socket,id=gns3-1,udp=192.168.1.165:10000,localaddr=0.0.0.0:10001
instead of the malformed:
-net udp,vlan=1,name=gns3-1,sport=10000,dport=10001,daddr=192.168.1.165
I'm guessing this is a GNS3 bug, and will report it as such...

Userlevel 5
Thanks for posting this and the update. Please Let me know what comes from the report with GNS3. I am actively interested in this flow.
Userlevel 3
Keep this stuff coming, Will, this is great. And let me know if anything here needs to be updated in the docs (even if it means calling out a GNS3 issue). Thanks!
Userlevel 2
Opened issue on GNS3-gui issue tracker: https://github.com/GNS3/gns3-gui/issues/663
Userlevel 2
OK, now I'm confused... I see in the doc http://docs.cumulusnetworks.com/display/VX/Using+GNS3+with+QEMU+and+KVM+Virtual+Machines that under the section entitled "KVM/QEMU from the Command Line" that the "-net udp,..." syntax was used there... So I tried the exact syntax of the command given on the first example there (excepting changing the path to the .qcow2 disk image) and this is what I get:

[gns3user@gns3server01 ~]$ qemu-system-x86_64 \  > -name "Cumulus VX-spine1" \
> -m 256 \
> -hda /home/gns3user/GNS3/images/CumulusVX-2.5.3-4eb681f3df86c478.qcow2 \
> -serial telnet:127.0.0.1:2003,server,nowait \
> -monitor tcp:127.0.0.1:55603,server,nowait \
> -net user,vlan=0,net=192.168.0.0/24,hostfwd=tcp::1403-:22 \
> -net none \
> -net nic,vlan=0,macaddr=00:00:ab:c9:df:00,model=virtio \
> -net nic,vlan=1,macaddr=00:00:ab:c9:df:01,model=virtio \
> -net udp,vlan=1,name=gns3-1,sport=10001,dport=10000,daddr=127.0.0.1 \
> -net nic,vlan=2,macaddr=00:00:ab:c9:df:02,model=virtio \
> -net udp,vlan=2,name=gns3-2,sport=10002,dport=10003,daddr=127.0.0.1 \
> -net nic,vlan=3,macaddr=00:00:ab:c9:df:03,model=virtio &
[1] 23040
[gns3user@gns3server01 ~]$
[gns3user@gns3server01 ~]$
[gns3user@gns3server01 ~]$ qemu-system-x86_64: -net udp,vlan=1,name=gns3-1,sport=10001,dport=10000,daddr=127.0.0.1: Invalid parameter 'udp'
[1]+ Exit 1 qemu-system-x86_64 -name "Cumulus VX-spine1" -m 256 -hda /home/gns3user/GNS3/images/CumulusVX-2.5.3-4eb681f3df86c478.qcow2 -serial telnet:127.0.0.1:2003,server,nowait -monitor tcp:127.0.0.1:55603,server,nowait -net user,vlan=0,net=192.168.0.0/24,hostfwd=tcp::1403-:22 -net none -net nic,vlan=0,macaddr=00:00:ab:c9:df:00,model=virtio -net nic,vlan=1,macaddr=00:00:ab:c9:df:01,model=virtio -net udp,vlan=1,name=gns3-1,sport=10001,dport=10000,daddr=127.0.0.1 -net nic,vlan=2,macaddr=00:00:ab:c9:df:02,model=virtio -net udp,vlan=2,name=gns3-2,sport=10002,dport=10003,daddr=127.0.0.1 -net nic,vlan=3,macaddr=00:00:ab:c9:df:03,model=virtio

Do I need some special version of QEMU that supports the "-net udp,..." syntax?

Userlevel 2
Will Dennis wrote:

OK, now I'm confused... I see in the doc

Userlevel 5
Will, have you seen this post?

https://community.gns3.com/ideas/1739

Userlevel 2
No, hadn't seen that... But note that the fellow in that post has the opposite problem of mine (new style QEMU "-netdev socket,..." command failing with udp error while mine seems to work...)

I see that Jeremy G. (main GNS3 dev) has labeled my GitHub issue #663 as "bug" and assigned it to himself, so let's see where that goes...
Userlevel 5
I did see that on the bug and he also marked it for the new release(1.4). Will keep eyes on it.
Userlevel 2
Jeremy wrote this comment in the GH issue:

Looks like recent version of Qemu do not understand the -udp parameter anymore. Need to investigate this more.

Alternatively, you can use virtio-net-pci network adapter type instead and not use the "legacy networking mode"

Is there a way we could work with the appropriate resource there at Cumulus to get the VX VM to allow SSH access without using "legacy networking mode"?
Userlevel 5
Will,

I have pinged the team internally and hopefully it will spark something. We will keep you updated. Thanks!
Userlevel 2
Great, thanks Scott!
Userlevel 2
Will Dennis wrote:

Great, thanks Scott!

Just got the definitive word from the main GNS3 dev (Jeremy G.) --

https://github.com/GNS3/gns3-gui/issues/663#issuecomment-145271210

So, the "-net udp,..." syntax was never valid in mainline QEMU[1]. You have to use the "-netdev socket,...,udp=..." syntax, which was added back in QEMU 1.1 to support UDP tunnels.

[1] It only worked if you patched the official QEMU source with a GNS3-curated patch set that supported the "-net udp,..." syntax and then built a custom QEMU; see http://www.braindeadprojects.com/blog/what/gns3-and-gentoo-fixing-qemu-networking/ for historical details.
Userlevel 2
OK, achievement unlocked... But first, let me explain the failure domains...

The VX "Using GNS3 with QEMU and KVM Virtual Machines" doc currently specifies in step 4 to "Check the 'Use the legacy networking mode' check box." This is mainly to allow SSH'ing to the VX box, which sadly is not set up for serial console access (this really should be enabled by default on VX - please!!) and so SSH is the only way in (not really, if you allow the QEMU console to be displayed, but for most folks this would be turned off.) However, due to the way GNS3 utilizes QEMU "legacy mode" and the history of GNS3 / QEMU integration, this causes the formation of incorrect syntax when forming the inter-VM links in GNS3.

To allow SSH access from the host, the legacy QEMU construct of:
-net user,vlan=0,net=192.168.0.0/24,hostfwd=tcp::1401-:22
is tied to the VX eth0 interface via the "vlan=0" parameter, which is also set on the QEMU NIC:
-net nic,vlan=0,macaddr=00:00:ab:f0:1e:00,model=virtio 
The "vlan=0" parameter implemented a virtual ethernet hub, in which any other -net interface with "vlan=0" was virtually connected to. So, the "-net user,..." line above allows the forwarding of traffic sent to the host TCP port 1401 to the VX NIC with the "vlan=0" tag, namely eth0, TCP port 22 (SSH.). Also, the "net=" in that line declares a IP subnet to which QEMU will provide DHCP service to, which also causes the eth0 NIC in VX to get an IP address (typically 192.168.0.15 from what I've seen.) And so, this is why (for this example) if one types "ssh -p 1401 cumulus@localhost" you get a SSH connection to the VX host, which it sees as coming in to 192.168.0.15:22, namely SSH on eth0.

The problem becomes that up until now, GNS3 in "legacy mode" would form the inter-switch connection in this format:
-net udp,vlan=1,name=gns3-1,sport=10001,dport=10000,daddr=127.0.0.1
which following the "vlan=" rule above, would be connected to the interface having "vlan=1", namely in this case swp1:
-net nic,vlan=1,macaddr=00:00:ab:c9:df:01,model=virtio
The problem being, as we've found out, that mainline QEMU never had a "-net udp,..." construct.

OK. But what if we UNcheck "legacy mode"?

Well, that makes the inter-switch links work! Because they are in a new format, namely:
-device virtio-net-pci,mac=00:00:ab:a7:ad:01,netdev=gns3-1 \
-netdev socket,id=gns3-1,udp=192.168.1.165:10001,localaddr=0.0.0.0:10000 \
Now they are tied together by a "netdev= / id=" keyname pairing, not a "vlan=" common parameter. And QEMU since v1.1 has supported UDP tunneling via the "udp=..." attribute in "-netdev socket,..." (and I think everyone now is [or should be] running QEMU 1.2 or better with their QEMU distro packages.)

HOWEVER - sadly, this breaks the magic SSH port forwarding, because of one little thing -- we would need a "netdev=" parameter tacked on to the end of the requisite eth0 NIC, and GNS3 has no way of doing that to match a parameter that gets tacked on via the "Advanced Settings > Additional Settings > Options" field that is provided in the GNS3 UI. So even if we did add the new style:
-netdev user,id=mgmt0,net=192.168.0.0/24,hostfwd=tcp::1501-:22
into that field, GNS3 does not have a way to add the matching
,netdev=mgmt0
to the end of the first "-device" line, which correlates to the VX VM's eth0 interface.

Man, this is getting loooong, and it's late - so begging your pardon, I'll add a "part 2" later today that explains how I engineered SSH access to VX, which is NOT running in "legacy mode"... stay tuned 🙂
Userlevel 2
OK, here's part2, which is itself a work in progress...

Thinking about the way I used to connect my GNS3 labs up to my host PC (the one running GNS3), I thought, hmmm... What about using a "tap" connection? So, I created a "tuntap" interface on my host server (I'm running GNS3 1.4b3 in split client/server mode; the GNS3 server is running on a Dell server in my lab that has plenty of CPU and RAM) via the following commands (all executed as the 'gns3user' user, who is the user who I'm running the GNS3 server under):
$ sudo ip tuntap add tap0 mode tap user gns3user
$ sudo ip addr add 192.168.0.254/24 dev tap0
$ sudo ip link set tap0 up
This gave me an up-and-running tap0 interface on the server that I could use in GNS3. I then dragged out the "Host" icon from the GNS3 sidebar (in the "End Devices" section) onto my topology, and connected the 1st VX VM's "Ethernet0" interface (corresponds to the eth0 interface in the VM) up to the Host's "nio_tap:tap0" interface. After this was done, my topology looked like this:


Then I fired up the "VX-test-1" VM, and using the serial connection (right-click icon, choose "Console" - to enable this see my comments at https://community.cumulusnetworks.com/cumulus/topics/enable ) I logged in and configured the eth0 IP address via /etc/network/interfaces then did an "sudo ifreload -a" to implement the change. I verified I had connectivity by pinging the link partner IP from both sides, and then just for fun, I got LLDP running on my GNS3 server (which is helpfully named 'gns3server'), verified that I saw the VX switch from the server, and tried SSH, which worked:
[gns3user@gns3server01 ~]$ sudo lldpcli show nei
--------------------------------------------------------------------------
LLDP neighbors:
--------------------------------------------------------------------------
Interface: enp6s0f0, via: LLDP, RID: 2, Time: 0 day, 00:16:26
Chassis:
ChassisID: mac 00:25:64:12:f2:b2
Port:
PortID: ifname g23
--------------------------------------------------------------------------
Interface: tap0, via: LLDP, RID: 4, Time: 0 day, 00:01:01
Chassis:
ChassisID: mac 00:00:ab:a7:ad:00
SysName: vx-test-1
SysDescr: Cumulus Linux version 2.5.3 running on QEMU Standard PC (i440FX + PIIX, 1996)
MgmtIP: 192.168.0.1
Capability: Bridge, off
Capability: Router, on
Port:
PortID: ifname eth0
PortDescr: eth0
--------------------------------------------------------------------------
[gns3user@gns3server01 ~]$
[gns3user@gns3server01 ~]$ ssh cumulus@192.168.0.1
cumulus@192.168.0.1's password:
Linux vx-test-1 3.2.65-1+deb7u2+cl2.5+2 #3.2.65-1+deb7u2+cl2.5+2 SMP Wed Jul 29 14:21:03 PDT 2015 x86_64
Welcome to Cumulus VX (TM)
Cumulus VX (TM) is an open-source LINUX (R) distribution. License files are included with every package installed in the system and can be viewed in the /usr/share/*/doc/copyright files.
The registered trademark Linux (R) is used pursuant to a sub-license from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis.
Last login: Fri Oct 9 01:25:07 2015
cumulus@vx-test-1$
Sweet victory was mine at last! 😛 Alas, though, it was short-lived... If one is good, *two* is better, right? So I modified my topology thusly:


Went in and configured the VX-test-2's eth0 IP, and then verified pings and LLDP again:
[root@gns3server01 ~]# lldpcli show nei
--------------------------------------------------------------------------
LLDP neighbors:
--------------------------------------------------------------------------
Interface: enp6s0f0, via: LLDP, RID: 2, Time: 0 day, 00:03:48
Chassis:
ChassisID: mac 00:25:64:12:f2:b2
Port:
PortID: ifname g23
--------------------------------------------------------------------------
Interface: tap0, via: LLDP, RID: 1, Time: 0 day, 00:03:52
Chassis:
ChassisID: mac 00:00:ab:a7:ad:00
SysName: vx-test-1
SysDescr: Cumulus Linux version 2.5.3 running on QEMU Standard PC (i440FX + PIIX, 1996)
MgmtIP: 192.168.0.1
Capability: Bridge, off
Capability: Router, on
Port:
PortID: ifname eth0
PortDescr: eth0
--------------------------------------------------------------------------
Interface: tap0, via: LLDP, RID: 3, Time: 0 day, 00:03:41
Chassis:
ChassisID: mac 00:00:ab:4c:10:00
SysName: vx-test-2
SysDescr: Cumulus Linux version 2.5.3 running on QEMU Standard PC (i440FX + PIIX, 1996)
MgmtIP: 192.168.0.2
Capability: Bridge, off
Capability: Router, on
Port:
PortID: ifname eth0
PortDescr: eth0
--------------------------------------------------------------------------
Looks good, right? So I try a SSH again:
[gns3user@gns3server01 ~]$ ssh cumulus@192.168.0.1
ssh: connect to host 192.168.0.1 port 22: Connection timed out
[gns3user@gns3server01 ~]$ ssh cumulus@192.168.0.2
ssh: connect to host 192.168.0.1 port 22: Connection timed out
Needless to say, I haz a sad... Not sure why this isn't working, but tomorrow's another day... At least some progress is being made 🙂

Reply