Introduction
In this article I want to show you how to start with a bare Cisco NX-OS QEMU image guest and configure it to be reachable via the network of the host system. In this approach I am going to use an ‘expect’ script that is going through the initial configuration dialogues of the console of NX-OS.
After having a readily set up system, you can continue with other expect-scripts or configure your Cisco switch with Ansible or the automation of your choice.
The approach is more raw than the approach with Vagrant described in an earlier article. This way works around all SSH and SCP problems that arise when using Vagrant. Besides that, you are not needing VirtualBox virtualization. You can save and load QEMU snapshots using the so-called QEMU monitor.
In the following I want to lead you through the steps to set up your first VM.
The steps are:
- Installing the Linux host dependencies.
- Downloading the Cisco NX-OS image.
- Downloading the wrapper scripts.
- Starting the NX-OS VM.
- Starting the expect script.
- Waiting for the expect script to set up the switch.
- Using the NX-OS switch via the exposed network ports.
Linux packages to install
You need to have the packages for QEMU in your Linux system. The following gives the Debian packages needed:
# apt-get install qemu-system-x86 qemu-utils ovmf expect netcat
I want to summarize for every package what it’s for:
qemu-system-x86: The x86 QEMU runtime.qemu-utils: Some utils like ‚qemu-img‘ which comes handy when creating a snapshot copy from another VM image.ovmf: An UEFI-enabled firmware for VMs that is needed to boot NX-OS properly.expect: A tool for automating interactive dialogue programs. In our case we’re automating the console-based setup of NX-OS with it.
Downloading a Cisco NX-OS image
You need a Cisco NX-OS image from Cisco. As a precondition you need a valid login and password and license at Cisco.
The images are usually called „Cisco Nexus 9000/3000 Virtual Switch for KVM“. A link for Nexus 9000v and release 9.3(10) is here:
https://software.cisco.com/download/home/286312239/type/282088129/release/9.3(10)
The downloaded file will have a name like nexus9300v.9.3.10.qcow2.
Downloading the wrapper scripts
In this github repo, there are multiple wrapper scripts showing the use of QEMU and also an expect script that is automating the initial setup of a NX-OS switch.
The following is an overview of the relevant files:
- qemu-snapshot-and-start.sh: Creates a snapshot from a (unaltered) Cisco Nexus image and starts from this snapshot. All data written is written to this snapshot.
- qemu-start-from-snapshot.sh: Starts a QEMU VM from a previously created snapshot (see above).
- expect.ex: An expect script that connects to an existing QEMU VM via Console TCP port and goes through the installation
Starting the NX-OS VM
The qemu-snapshot-and-start.sh script requires you to pass the downloaded NX-OS image as an environment varialbe called $SOURCE_IMAGE.
When you have done so, you can call the script.
The steps will then be the following:
Creating an image
Create a copy-on-write image of the input image:
qemu-img create -f qcow2 -b ${SOURCE_IMAGE} ${SNAPSSHOT_IMAGE}
The reason for this is to not alter the ‚clean‘ original image. Besides that, images created using this approach are copy-on-write images, meaning that you don’t need to duplicate the whole VM image. The backing image is required for the VM to work.
Spawning QEMU
The following is the command line the script uses to spawn QEMU.
I want to give you the command and then discuss every option:
qemu-system-x86_64 -smp 2 -enable-kvm \
-nographic \
-bios /usr/share/qemu/OVMF.fd \
-m 8192 \
-serial telnet:localhost:2023,server=on,wait=off \
-device ahci,id=ahci0,bus=pci.0 -drive file=${SNAPSSHOT_IMAGE},if=none,id=drive-sata-disk0,id=drive-sata-disk0,format=qcow2 \
-device ide-hd,bus=ahci0.0,drive=drive-sata-disk0,id=drive-sata-disk0,bootindex=1 \
-netdev user,id=net0,hostfwd=tcp::3022-:22,hostfwd=tcp::3080-:80,hostfwd=tcp::3443-:443 \
-device e1000,netdev=net0,mac=aa:bb:cc:dd:ee:ff,multifunction=on,romfile=,bootindex=3
Now let’s discuss every option:
| Option | Meaning |
| -smp 2 | Have 2 CPU cores. |
| -enable-kvm | Enable KVM virtualization (acceleration). |
| -nographic | Do not show the VGA console of the switch. NX-OS will to serial console in the boot process very early, so the VGA console will be useless very early. |
| -bios /usr/share/qemu/OVMF.fd | Specify the OVMF bios that is needed by NX-OS to boot. |
| -m 8192 | Start with 8 GB of RAM. |
| -serial telnet:localhost:2023,server=on,wait=off | Bind the console UART to the hosts TCP port 2023. |
| -device ahci,id=ahci0,bus=pci.0 | Register a SATA device in AHCI mode |
| -drive file=${SNAPSSHOT_IMAGE},if=none,id=drive-sata-disk0,id=drive-sata-disk0,format=qcow2 | Register the first hard disk to the image |
| -device ide-hd,bus=ahci0.0,drive=drive-sata-disk0,id=drive-sata-disk0,bootindex=1 | Bind the previously registered HDD to the first IDE |
| -netdev user,id=net0, hostfwd=tcp::3022-:22, hostfwd=tcp::3080-:80, hostfwd=tcp::3443-:443 | Register net0 with multiple TCP sockets from host to guest |
| -device e1000,netdev=net0, mac=aa:bb:cc:dd:ee:ff, multifunction=on,romfile=,bootindex=3 | Bind the net0 netdev to a e1000 NIC. Lower the bootindex so OVMF is not trying netboot. |
The above command will map the following TCP ports from the host to the guest:
| Host | Guest | Meaning |
| 2023 | console | The serial console port |
| 3022 | 22 | SSH port |
| 3080 | 80 | HTTP port (unencrypted, for tcpdump) |
| 3443 | 443 | HTTPS port (encrypted) |
QEMU commands
Useful QEMU commands are:
quit: Quit the VM.savevm $NAME: Save the VM state (HDD + MEM + CPU) to a snapshot called $NAME in the HDD image.loadvm $NAME: Load the VM state from a snapshot called $NAME from the HDD image.
Automating QEMU
QEMU has a console to modify the running VM.
By adding a monitor TCP port, you can access the QEMU console via TCP and send QEMU commands to it (see above). This can be done by adding this to the QEMU command line:
-monitor telnet::45454,server,nowait
Starting the expect script
When QEMU is running and accepting TCP connections to the console port, a bare NX-OS switch will boot and wait for configuration inputs.
This is the point of time where the expect script kicks in.
$ ./expect.ex .... switch(config)# int mgmt0 int mgmt0 switch(config-if)# ip address dhcp ip address dhcp Warning: Disable ip dhcp relay if enabled. switch(config-if)# exit exit switch(config)# copy running-config startup-config copy running-config startup-config [########################################] 100% Copy complete, now saving to disk (please wait)... Copy complete. switch(config)# exit exit switch# 12:44:28 INFO OK: Expect script finished with success. Expect script took 255 seconds.
It connects to the NX-OS VM using the console port. It does then the following:
- Specify boot image if stuck in boot loader (image name may need to be adjusted).
- Abort Power On Auto Provisionig.
- Deny to enfore secure passwords.
- Enter user/password of the ‚admin‘ user.
- Not enter the basic config dialog.
- Login with the previously defined admin user/password.
- Configure the boot image.
- Activate DHCP on interface mgmt0 so the QEMU DHCP service can assign an IP address to the switch.
- Persist all changes so far to the startup-config.
The whole procedure will take around 4.5 minutes. Because of this it really pays off to take a snapshot of the NX-OS VM once you are done so you can restore the snapshot in seconds.
The expect script is using green-colored lines for informational messages and red-colored lines for error messages.
When the expect script is finished, it will print
OK: Expect script finished with success. Expect script took ... seconds.
Using the NX-OS switch via the exposed network ports
The NX-OS switch is now booted and reachable from the hosts network.
You can log in with a normal login user or an admin user.
| Username | Password |
| admin | admin |
You can connect to the switch either using telnet and a console port, or SSH:
$ ssh -oPubKeyAuthentication=false -p3022 admin@localhost
ssh -oPubKeyAuthentication=false -p3022 admin@localhost
The authenticity of host '[localhost]:3022 ([127.0.0.1]:3022)' can't be established.
RSA key fingerprint is SHA256:FronHu+9JxcRpCOvGcwpKEF+MZdfWoS94cM6MhcGghg.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[localhost]:3022' (RSA) to the list of known hosts.
User Access Verification
Password:
Bad terminal type: "xterm-256color". Will assume vt100.
Cisco NX-OS Software
Copyright (c) 2002-2020, Cisco Systems, Inc. All rights reserved.
Nexus 9000v software ("Nexus 9000v Software") and related documentation,
files or other reference materials ("Documentation") are
the proprietary property and confidential information of Cisco
Systems, Inc. ("Cisco") and are protected, without limitation,
pursuant to United States and International copyright and trademark
laws in the applicable jurisdiction which provide civil and criminal
penalties for copying or distribution without Cisco's authorization.
Any use or disclosure, in whole or in part, of the Nexus 9000v Software
or Documentation to any third party for any purposes is expressly
prohibited except as otherwise authorized by Cisco in writing.
The copyrights to certain works contained herein are owned by other
third parties and are used and distributed under license. Some parts
of this software may be covered under the GNU Public License or the
GNU Lesser General Public License. A copy of each such license is
available at
http://www.gnu.org/licenses/gpl.html and
http://www.gnu.org/licenses/lgpl.html
***************************************************************************
* Nexus 9000v is strictly limited to use for evaluation, demonstration *
* and NX-OS education. Any use or disclosure, in whole or in part of *
* the Nexus 9000v Software or Documentation to any third party for any *
* purposes is expressly prohibited except as otherwise authorized by *
* Cisco in writing. *
***************************************************************************
switch#
Summary
In this article I’ve shown how to use QEMU with KVM.
VM snapshots are stored in the QCOW2 image file and can be restored later on per command towards the QEMU monitor port. This saves a lot of time when in a testing environment.
The approach feels raw from the first view, but gives you a lot of control over your VM.
Compared to the vagrant approach discussed in an article before, this approach spawns less processes, does not mess around with the switches default shells and has more control over the image files.