This is a step-by-step instruction on how to get started with virtualization with Vagrant, VirtualBox and Cisco NX-OS images. This comes handy when developing network automation approaches and testing them in a test environment with no cost of real hardware.
An article showing a more raw QEMU/KVM based approach can be found here.
Vagrant is a virtualization and automation frontend for multiple virtualization backends like virtualbox, libvirt, vmware. It comes with a image management and scripting approach called ‚Vagrantfile‘.
Please note that for most vagrant operations to work you need to have the Vagrantfile in your current working directory.
Cisco provides Vagrant images of their Nexus 9300/9500 virtual switch series.
In summary, you are needing these steps:
- Install OS dependencies
- Download Cisco NX-OS box image
- Import Cisco NX-OS box image
- Write Vagrantfile
- Start VM with
- If needed, do provisioning with
- Stop VM with
vagrant destroy -f
Linux packages to install
You need to have the packages for vagrant and virtualbox in your Linux system. The following gives the Debian packages needed:
# apt-get install vagrant virtualbox
Downloading a Cisco NX-OS image
You need a Cisco NX-OS image from Cisco. As a precondition you need a valid login and password and license at Cisco.
The images are usually called „Cisco Nexus 9000/3000 Virtual Switch for Vagrant“. A link for Nexus 9000v and release 9.3(10) is here:
The downloaded file will have a name like
Importing the box to Vagrant
To import the downloaded box-image into Vagrant, issue the command
$ vagrant box add nexus9300v.9.3.5 ~/Downloads/nexus9300v.9.3.10.box –force
This will import the file towards the logical name
nxos/9.3.10. This is the name of the „config.vm.box“ you’ll need later in the Vagrantfile.
Afterwards you can see the box in your local repository:
$ vagrant box list debian/bullseye64 (virtualbox, 11.20220718.1) nexus9300v.9.3.10 (virtualbox, 0) nexus9300v.9.3.5 (virtualbox, 0) ubuntu/trusty64 (virtualbox, 20190514.0.0)
Creating a Vagrantfile for the box
Create a new text file called Vagrantfile defining your setup. This defines your whole VM setup and how to configure your NOS.
The following is a table of mapped TCP/UDP ports of the Vagrant file shown in the next section:
|Console||TCP||UART1||2023||Console port, not directly assigned in Vagrantfile|
|SSH||TCP||22||3022||SSH service, also used for vagrant provisioning|
|HTTP||TCP||80||3080||HTTP service, not used anymore by current NX-OS|
|HTTPS||TCP||443||3443||HTTPS service for „feature nxapi“|
|SNMP||UDP||161||3161||SNMP UDP port forward|
The following is a Vagrantfile with the guest TCP/UDP ports mentioned above mapped to host ports. A Vagrantfile is also given on this Github repo.
# -*- mode: ruby -*- # vi: set ft=ruby : Vagrant.configure("2") do |config| # name of the box as in "vagrant box add" config.vm.box = "nexus9300v.9.3.5" # foward ssh port config.vm.network :forwarded_port, guest: 22, host: 3022, id: 'ssh', auto_correct: true # foward http port not needed anymore, current NXOS uses TCP 443 for feature nxapi #config.vm.network :forwarded_port, guest: 80, host: 3080, id: 'http', auto_correct: true # enable "feature nxapi", browse to "https://localhost:3443/" config.vm.network :forwarded_port, guest: 443, host: 3443, id: 'https', auto_correct: true # netconf service config.vm.network :forwarded_port, guest: 830, host: 3830, id: 'netconf', auto_correct: true # forward UDP snmp port config.vm.network :forwarded_port, guest: 161, host: 3161, protocol: 'udp', id: 'snmp', auto_correct: true config.ssh.shell = "run bash" config.vm.provider "virtualbox" do |vb| # vb.gui = true vb.cpus = "2" vb.memory = "8192" vb.customize ["modifyvm", :id, "--firmware", "efi"] # telnet console, usually on tcp 2023 vb.customize ["modifyvm", :id, "--uart1", "0x3F8", "4"] end end
Starting up your VM
Change the working directory towards your Vagrantfile definition.
$ vagrant up Bringing machine 'default' up with 'virtualbox' provider... ==> default: Importing base box 'nexus9300v.9.3.5'... ==> default: Matching MAC address for NAT networking... ==> default: Setting the name of the VM: nexus_default_1661767270999_61318 ==> default: Clearing any previously set network interfaces... ==> default: Preparing network interfaces based on configuration... default: Adapter 1: nat default: Adapter 2: intnet ==> default: Forwarding ports... default: 22 (guest) => 3022 (host) (adapter 1) default: 80 (guest) => 3080 (host) (adapter 1) default: 443 (guest) => 3443 (host) (adapter 1) default: 830 (guest) => 3830 (host) (adapter 1) ==> default: Running 'pre-boot' VM customizations... ==> default: Booting VM... ==> default: Waiting for machine to boot. This may take a few minutes... default: SSH address: 127.0.0.1:3022 default: SSH username: vagrant default: SSH auth method: private key
NX-OS is now starting up. After some time the vagrant output will print something about an „insecure key“ (which is standard in vagrant)
default: Vagrant insecure key detected. Vagrant will automatically replace default: this with a newly generated keypair for better security. default: default: Inserting generated public key within guest... ==> default: Attempting graceful shutdown of VM... The following SSH command responded with a non-zero exit status. Vagrant assumes that this means the command failed! shutdown -h now Stdout from the command: Stderr from the command:
Using your VM
You can now start up the VirtualBox user interface and take a look at your VM. Don’t expect too much from the VGA output, NX-OS will print almost nothing there and allow no login via the VGA console. See
Pre-defined users and their passwords
After booting, you can log in to the UART1 console or SSH-forwarded port using the following users that are preconfigured in the NX-OS box:
|admin||admin||Standard Cisco admin user.|
|vagrant||vagrant||Standard vagrant automation user for SSH. Is used by „vagrant ssh“ (see below).|
Usage via console UART1
You can connect towards the UART1 to access the console and see NX-OS booting:
$ telnet localhost 2023 Trying ::1... Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. cat: /sys/module/klm_cctrli*/parameters/cctrl_hw_rev_maj*: No such file or directory cp: cannot stat '/nxos/tmp/vdc_1_sup1': No such file or directory rm: cannot remove '/isan/bin/bios_imgs/*': No such file or directory rm: cannot remove '/isan/lib/libtahusdcli.so': No such file or directory Linking n9k flash devices creating flash devices BOOT_DEV= sda mknod: '/dev/sda7': File exists Replacing inittab for N9Kv pEoR node - vsup INIT: version 2.88 booting Unsquashing rootfs ... Total size needed in bootflash is 108588 check bootflash : OK Total size needed in bootflash is 41032 check bootflash : OK No TOR virtual chassis Enabling 8250 serial driver spurious INTs workaround Installing isan procfs isan_proc_name=isan26... done. ...
Usage via „vagrant ssh“
Vagrant has a pre-defined wiring towards the VMs ssh port for automation purposes. You can access the ssh port after VM boot using the vagrant command
$ vagrant ssh Cisco NX-OS Software Copyright (c) 2002-2020, Cisco Systems, Inc. All rights reserved. Nexus 9000v software ("Nexus 9000v Software") and related documentation, files or other reference materials ("Documentation") are the proprietary property and confidential information of Cisco Systems, Inc. ("Cisco") and are protected, without limitation, pursuant to United States and International copyright and trademark laws in the applicable jurisdiction which provide civil and criminal penalties for copying or distribution without Cisco's authorization. Any use or disclosure, in whole or in part, of the Nexus 9000v Software or Documentation to any third party for any purposes is expressly prohibited except as otherwise authorized by Cisco in writing. The copyrights to certain works contained herein are owned by other third parties and are used and distributed under license. Some parts of this software may be covered under the GNU Public License or the GNU Lesser General Public License. A copy of each such license is available at http://www.gnu.org/licenses/gpl.html and http://www.gnu.org/licenses/lgpl.html *************************************************************************** * Nexus 9000v is strictly limited to use for evaluation, demonstration * * and NX-OS education. Any use or disclosure, in whole or in part of * * the Nexus 9000v Software or Documentation to any third party for any * * purposes is expressly prohibited except as otherwise authorized by * * Cisco in writing. * *************************************************************************** Nexus9000v#
Provisioning your VM
Usually Vagrant will execute the provisioning parts of your VM. For NX-OS, there are usually some problems with the vagrant startup. You usually need to call „vagrant provision“ by hand if you have provisioning parts in your Vagrantfile. Here’s an example of a Ansible provisioning:
$ vagrant provision ==> nxos1: Running provisioner: ansible... nxos1: Running ansible-playbook... PLAY [Provision NX-OS Devices] ************************************************* TASK [Pause to complete boot] ************************************************** [WARNING]: ansible-pylibssh not installed, falling back to paramiko Pausing for 5 seconds (ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
Stopping your VM
The VM can be stopped in multiple ways. The safest way is to destroy the VM:
$ vagrant destroy -f ==> default: Forcing shutdown of VM... ==> default: Destroying VM and associated drives...
Snapshotting the VM state
Snapshotting the state of the VM and restoring it is a way to quickly roll back the VM to a defined state.
This only works while the VM is running. As soon as the VM is destroyed, also the snapshots go away.
Saving a snapshot
The following example saves a snapshot called „aftersnap“:
$ vagrant snapshot save default aftersnap ==> default: Snapshotting the machine as 'aftersnap'... ==> default: Snapshot saved! You can restore the snapshot at any time by ==> default: using `vagrant snapshot restore`. You can delete it using ==> default: `vagrant snapshot delete`.
Listing the snapshots
The following lists all snapshots. In this example there were two snapshots saved:
$ vagrant snapshot list ==> default: aftersnap init
Restoring a snapshot
The following restores a snapshot from disk and replaces the current VM state with the snapshotted state
$ vagrant snapshot restore default aftersnap2 ==> default: Forcing shutdown of VM... ==> default: Restoring the snapshot 'aftersnap2'... ==> default: Resuming suspended VM... ==> default: Booting VM... ==> default: Waiting for machine to boot. This may take a few minutes... default: SSH address: 127.0.0.1:3022 default: SSH username: vagrant default: SSH auth method: private key The configured shell (config.ssh.shell) is invalid and unable to properly execute commands. The most common cause for this is using a shell that is unavailable on the system. Please verify you're using the full path to the shell and that the shell is executable by the SSH user.
Virtualbox can’t be installed
You might need to add another repository to your Debian system. See here.
Isn’t VirtualBox having licensing issues?
VirtualBox itself is licensed under GPL v2. VirtualBox extension pack is under a Oracle license, but not needed for Vagrant.
What’s the boot time for a NX-OS switch?
The boot time is about 4:30 minutes on my Dell Latitude 5501.
Can the boot time be reduced?
One way to not have the boot time for every test case, I recommend keeping the VM running and just restoring VM snapshots.
VM snapshot restore costs only about 40s in contrast to 270s full boot (6 times faster).
What are the memory requirements?
NX-OS needs at least 6 GB of RAM according to many internet posts, ideally it requires 8 GB RAM. This means you need a lot of RAM.
What’s already configured in the VM?
The Vagrant box from Cisco has already some things pre-configured for the Vagrant setup to work:
- User vagrant for „vagrant ssh“
- Interface „mgmt0“ is up and has an IP address from VirtualBox DHCP?
Why does shell provisioning not work?
Vagrant shell provisioning does not work because of several reasons:
Secure copy: Access Denied (Net::SCP::Error)
- Problem: Vagrant is using a scp-based mechanism and scp is not enabled per default.
- Solution: „feature scp“ needs to be enabled.
scp: /bootflash/tmp/vagrant-shell: No such file or directory (Net::SCP::Error)
- Problem: The filesystem path /bootflash/tmp is not existing.
- Solution: You can override vagrants path with the ‚upload_path‘ option.
config.vm.provision "shell", upload_path: "tmpfile", inline: "echo bash"
The SSH command responded with a non-zero exit status
- Problem: Changing the uploaded file attributes fails because vagrant is logging in with a Cisco CLI, not a bash
INFO ssh: Execute: chmod +x 'tmpfile' && tmpfile (sudo=true)
DEBUG ssh: Exit status: 20
- Solution: Unknown
The topic can be troubleshooted using
vagrant provision --debug
Cisco proposes to use Ansible provisioning which is more complex.
Is there a demo on how to do Ansible provisioning?
Yes, there is a sample repo here.
Is there a way to do a simple shell provisioning?
You can add a simple sequence like this to your Vagrantfile:
if VAGRANT_COMMAND = "up" config.ssh.shell = "conf t ; hostname helloworld ; copy r s" end
Is it possible to simulate multiple switches / bigger topology?
There are Vagrantfile examples defining bigger topologies, but I did not test them. Sample: https://github.com/hpreston/vagrant_net_prog/blob/master/nx-os/two-switches/Vagrantfile
Can SNMP be also simulated?
Yes, you only need to configure SNMP on your switch and specify the fowarded UDP port number in the command line (here 3161)
snmpwalk -v2c -cpublic localhost:3161
In this article the basics of Vagrant virtualization of a Cisco NX-OS switch is shown.
There are some challenges when it comes to Vagrant operations that require a full-blown Linux system, but find a network device. These are SSH-based automation and SCP. I found automation to not work reliably with simple Vagrant provision commands, but to work with Ansible scripting.
NX-OS startup time is slow for testing environments that require a clean test target. Vagrant offers a snapshot mechanism to reset your VM towards a named well-defined state. The snapshot mechanism of Vagrant is heavily dependant on the virtualization solution it uses. The location of your snapshot file(s) is defined by VirtualBox.
With Ansible scripting provisioning, automation worked pretty well.
In the following I want to give you pointers to further information:
- Vagrant file sample repo: https://github.com/sfuhrm/vagrant-cisco-nexus
- Cisco Live presentation: https://www.ciscolive.com/c/dam/r/ciscolive/us/docs/2018/pdf/DEVNET-1365.pdf
- Github repo with different Vagrantfile examples: https://github.com/hpreston/vagrant_net_prog/tree/master/nx-os