Introduction

This is a step-by-step instruction on how to get started with virtualization with Vagrant, VirtualBox and Cisco NX-OS images. This comes handy when developing network automation approaches and testing them in a test environment with no cost of real hardware.

An article showing a more raw QEMU/KVM based approach can be found here.

Vagrant is a virtualization and automation frontend for multiple virtualization backends like virtualbox, libvirt, vmware. It comes with a image management and scripting approach called ‚Vagrantfile‘.

Please note that for most vagrant operations to work you need to have the Vagrantfile in your current working directory.

Cisco provides Vagrant images of their Nexus 9300/9500 virtual switch series.

In summary, you are needing these steps:

1. Install OS dependencies
2. Download Cisco NX-OS box image
3. Import Cisco NX-OS box image
4. Write Vagrantfile
5. Start VM with vagrant up
6. If needed, do provisioning with vagrant provision
7. Stop VM with vagrant destroy -f

Setup

Linux packages to install

You need to have the packages for vagrant and virtualbox in your Linux system. The following gives the Debian packages needed:

# apt-get install vagrant virtualbox

Downloading a Cisco NX-OS image

You need a Cisco NX-OS image from Cisco. As a precondition you need a valid login and password and license at Cisco.

The images are usually called „Cisco Nexus 9000/3000 Virtual Switch for Vagrant“. A link for Nexus 9000v and release 9.3(10) is here:

https://software.cisco.com/download/home/286312239/type/282088129/release/9.3(10)

The downloaded file will have a name like nexus9300v.9.3.10.box.

Importing the box to Vagrant

To import the downloaded box-image into Vagrant, issue the command

$ vagrant box add nexus9300v.9.3.5 ~/Downloads/nexus9300v.9.3.10.box –force

This will import the file towards the logical name nxos/9.3.10. This is the name of the „config.vm.box“ you’ll need later in the Vagrantfile.

Afterwards you can see the box in your local repository:

$ vagrant box list
debian/bullseye64 (virtualbox, 11.20220718.1)
nexus9300v.9.3.10 (virtualbox, 0)
nexus9300v.9.3.5  (virtualbox, 0)
ubuntu/trusty64   (virtualbox, 20190514.0.0)

Creating a Vagrantfile for the box

Create a new text file called Vagrantfile defining your setup. This defines your whole VM setup and how to configure your NOS.

The following is a table of mapped TCP/UDP ports of the Vagrant file shown in the next section:

Service	Proto	Guest	Host	Description
Console	TCP	UART1	2023	Console port, not directly assigned in Vagrantfile
SSH	TCP	22	3022	SSH service, also used for vagrant provisioning
HTTP	TCP	80	3080	HTTP service, not used anymore by current NX-OS
HTTPS	TCP	443	3443	HTTPS service for „feature nxapi“
NetConf	TCP	830	3830	NetConf service
SNMP	UDP	161	3161	SNMP UDP port forward

Vagrantfile

The following is a Vagrantfile with the guest TCP/UDP ports mentioned above mapped to host ports. A Vagrantfile is also given on this Github repo.

# -*- mode: ruby -*-
# vi: set ft=ruby :
 
Vagrant.configure("2") do |config|
 
  # name of the box as in "vagrant box add"
  config.vm.box = "nexus9300v.9.3.5"
 
  # foward ssh port
  config.vm.network :forwarded_port, guest: 22, host: 3022, id: 'ssh', auto_correct: true
  # foward http port not needed anymore, current NXOS uses TCP 443 for feature nxapi
  #config.vm.network :forwarded_port, guest: 80, host: 3080, id: 'http', auto_correct: true
  # enable "feature nxapi", browse to "https://localhost:3443/"
  config.vm.network :forwarded_port, guest: 443, host: 3443, id: 'https', auto_correct: true
  # netconf service
  config.vm.network :forwarded_port, guest: 830, host: 3830, id: 'netconf', auto_correct: true
  # forward UDP snmp port
  config.vm.network :forwarded_port, guest: 161, host: 3161, protocol: 'udp', id: 'snmp', auto_correct: true
  config.ssh.shell = "run bash"
  config.vm.provider "virtualbox" do |vb|
#     vb.gui = true
     vb.cpus = "2"
     vb.memory = "8192"
     vb.customize ["modifyvm", :id, "--firmware", "efi"]
     # telnet console, usually on tcp 2023
     vb.customize ["modifyvm", :id, "--uart1", "0x3F8", "4"]
    end
end

Starting up your VM

Change the working directory towards your Vagrantfile definition. 

$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'nexus9300v.9.3.5'...
==> default: Matching MAC address for NAT networking...
==> default: Setting the name of the VM: nexus_default_1661767270999_61318
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
    default: Adapter 2: intnet
==> default: Forwarding ports...
    default: 22 (guest) => 3022 (host) (adapter 1)
    default: 80 (guest) => 3080 (host) (adapter 1)
    default: 443 (guest) => 3443 (host) (adapter 1)
    default: 830 (guest) => 3830 (host) (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:3022
    default: SSH username: vagrant
    default: SSH auth method: private key

NX-OS is now starting up. After some time the vagrant output will print something about an „insecure key“ (which is standard in vagrant)

    default: Vagrant insecure key detected. Vagrant will automatically replace
    default: this with a newly generated keypair for better security.
    default:
    default: Inserting generated public key within guest...
==> default: Attempting graceful shutdown of VM...
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!
 
shutdown -h now
 
Stdout from the command:
 
 
 
Stderr from the command:

Using your VM

You can now start up the VirtualBox user interface and take a look at your VM. Don’t expect too much from the VGA output, NX-OS will print almost nothing there and allow no login via the VGA console. See

Pre-defined users and their passwords

After booting, you can log in to the UART1 console or SSH-forwarded port using the following users that are preconfigured in the NX-OS box:

User	Password	Purpose
admin	admin	Standard Cisco admin user.
vagrant	vagrant	Standard vagrant automation user for SSH. Is used by „vagrant ssh“ (see below).

Usage via console UART1

You can connect towards the UART1 to access the console and see NX-OS booting:

$ telnet localhost 2023
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
cat: /sys/module/klm_cctrli*/parameters/cctrl_hw_rev_maj*: No such file or directory
cp: cannot stat '/nxos/tmp/vdc_1_sup1': No such file or directory
rm: cannot remove '/isan/bin/bios_imgs/*': No such file or directory
rm: cannot remove '/isan/lib/libtahusdcli.so': No such file or directory
 
Linking n9k flash devices
creating flash devices BOOT_DEV= sda
mknod: '/dev/sda7': File exists
Replacing inittab for N9Kv pEoR node - vsup
INIT: version 2.88 booting
Unsquashing rootfs ...
Total size needed in bootflash is 108588
check bootflash : OK
Total size needed in bootflash is 41032
check bootflash : OK
No TOR virtual chassis
Enabling 8250 serial driver spurious INTs workaround
Installing isan procfs isan_proc_name=isan26... done.
...

Usage via „vagrant ssh“

Vagrant has a pre-defined wiring towards the VMs ssh port for automation purposes. You can access the ssh port after VM boot using the vagrant command

$ vagrant ssh
 
Cisco NX-OS Software
Copyright (c) 2002-2020, Cisco Systems, Inc. All rights reserved.
Nexus 9000v software ("Nexus 9000v Software") and related documentation,
files or other reference materials ("Documentation") are
the proprietary property and confidential information of Cisco
Systems, Inc. ("Cisco") and are protected, without limitation,
pursuant to United States and International copyright and trademark
laws in the applicable jurisdiction which provide civil and criminal
penalties for copying or distribution without Cisco's authorization.
 
Any use or disclosure, in whole or in part, of the Nexus 9000v Software
or Documentation to any third party for any purposes is expressly
prohibited except as otherwise authorized by Cisco in writing.
The copyrights to certain works contained herein are owned by other
third parties and are used and distributed under license. Some parts
of this software may be covered under the GNU Public License or the
GNU Lesser General Public License. A copy of each such license is
available at
http://www.gnu.org/licenses/gpl.html and
http://www.gnu.org/licenses/lgpl.html
***************************************************************************
*  Nexus 9000v is strictly limited to use for evaluation, demonstration   *
*  and NX-OS education. Any use or disclosure, in whole or in part of     *
*  the Nexus 9000v Software or Documentation to any third party for any   *
*  purposes is expressly prohibited except as otherwise authorized by     *
*  Cisco in writing.                                                      *
***************************************************************************
Nexus9000v#

Provisioning your VM

Usually Vagrant will execute the provisioning parts of your VM. For NX-OS, there are usually some problems with the vagrant startup. You usually need to call „vagrant provision“ by hand if you have provisioning parts in your Vagrantfile. Here’s an example of a Ansible provisioning:

$ vagrant provision
==> nxos1: Running provisioner: ansible...
    nxos1: Running ansible-playbook...
 
PLAY [Provision NX-OS Devices] *************************************************
 
TASK [Pause to complete boot] **************************************************
[WARNING]: ansible-pylibssh not installed, falling back to paramiko
Pausing for 5 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)

Stopping your VM

The VM can be stopped in multiple ways. The safest way is to destroy the VM:

$ vagrant destroy -f
==> default: Forcing shutdown of VM...
==> default: Destroying VM and associated drives...

Snapshotting the VM state

Snapshotting the state of the VM and restoring it is a way to quickly roll back the VM to a defined state.

This only works while the VM is running. As soon as the VM is destroyed, also the snapshots go away.

Saving a snapshot

The following example saves a snapshot called „aftersnap“:

$ vagrant snapshot save default aftersnap
==> default: Snapshotting the machine as 'aftersnap'...
==> default: Snapshot saved! You can restore the snapshot at any time by
==> default: using `vagrant snapshot restore`. You can delete it using
==> default: `vagrant snapshot delete`.

Listing the snapshots

The following lists all snapshots. In this example there were two snapshots saved:

$ vagrant snapshot list
==> default:
aftersnap
init

Restoring a snapshot

The following restores a snapshot from disk and replaces the current VM state with the snapshotted state

$ vagrant snapshot restore default aftersnap2
==> default: Forcing shutdown of VM...
==> default: Restoring the snapshot 'aftersnap2'...
==> default: Resuming suspended VM...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:3022
    default: SSH username: vagrant
    default: SSH auth method: private key
The configured shell (config.ssh.shell) is invalid and unable
to properly execute commands. The most common cause for this is
using a shell that is unavailable on the system. Please verify
you're using the full path to the shell and that the shell is
executable by the SSH user.

FAQ

Virtualbox can’t be installed

You might need to add another repository to your Debian system. See here.

Isn’t VirtualBox having licensing issues?

VirtualBox itself is licensed under GPL v2. VirtualBox extension pack is under a Oracle license, but not needed for Vagrant.

What’s the boot time for a NX-OS switch?

The boot time is about 4:30 minutes on my Dell Latitude 5501.

Can the boot time be reduced?

One way to not have the boot time for every test case, I recommend keeping the VM running and just restoring VM snapshots.

VM snapshot restore costs only about 40s in contrast to 270s full boot (6 times faster).

What are the memory requirements?

NX-OS needs at least 6 GB of RAM according to many internet posts, ideally it requires 8 GB RAM. This means you need a lot of RAM.

What’s already configured in the VM?

The Vagrant box from Cisco has already some things pre-configured for the Vagrant setup to work:

• User vagrant for „vagrant ssh“
• Interface „mgmt0“ is up and has an IP address from VirtualBox DHCP?

Why does shell provisioning not work?

Vagrant shell provisioning does not work because of several reasons:

• Secure copy: Access Denied (Net::SCP::Error)
  • Problem: Vagrant is using a scp-based mechanism and scp is not enabled per default.
  • Solution: „feature scp“ needs to be enabled.
• scp: /bootflash/tmp/vagrant-shell: No such file or directory (Net::SCP::Error)
  • Problem: The filesystem path /bootflash/tmp is not existing.
  • Solution: You can override vagrants path with the ‚upload_path‘ option.

config.vm.provision "shell", upload_path: "tmpfile", inline: "echo bash"

• The SSH command responded with a non-zero exit status
  • Problem: Changing the uploaded file attributes fails because vagrant is logging in with a Cisco CLI, not a bash

INFO ssh: Execute: chmod +x 'tmpfile' && tmpfile (sudo=true)
DEBUG ssh: Exit status: 20

• Solution: Unknown

The topic can be troubleshooted using

$ vagrant provision --debug

Cisco proposes to use Ansible provisioning which is more complex.

Is there a demo on how to do Ansible provisioning?

Yes, there is a sample repo here.

Is there a way to do a simple shell provisioning?

You can add a simple sequence like this to your Vagrantfile:

if VAGRANT_COMMAND = "up"
  config.ssh.shell = "conf t ; hostname helloworld ; copy r s"
end

Is it possible to simulate multiple switches / bigger topology?

There are Vagrantfile examples defining bigger topologies, but I did not test them. Sample: https://github.com/hpreston/vagrant_net_prog/blob/master/nx-os/two-switches/Vagrantfile

Can SNMP be also simulated?

Yes, you only need to configure SNMP on your switch and specify the fowarded UDP port number in the command line (here 3161)

$ snmpwalk -v2c -cpublic localhost:3161

Summary

In this article the basics of Vagrant virtualization of a Cisco NX-OS switch is shown.

There are some challenges when it comes to Vagrant operations that require a full-blown Linux system, but find a network device. These are SSH-based automation and SCP. I found automation to not work reliably with simple Vagrant provision commands, but to work with Ansible scripting.

NX-OS startup time is slow for testing environments that require a clean test target. Vagrant offers a snapshot mechanism to reset your VM towards a named well-defined state. The snapshot mechanism of Vagrant is heavily dependant on the virtualization solution it uses. The location of your snapshot file(s) is defined by VirtualBox.

With Ansible scripting provisioning, automation worked pretty well.

Further reading

In the following I want to give you pointers to further information:

• Vagrant file sample repo: https://github.com/sfuhrm/vagrant-cisco-nexus
• Cisco Live presentation: https://www.ciscolive.com/c/dam/r/ciscolive/us/docs/2018/pdf/DEVNET-1365.pdf
• Github repo with different Vagrantfile examples: https://github.com/hpreston/vagrant_net_prog/tree/master/nx-os