Archiv der Kategorie: Linux

Network Device Simulation for Cisco Nexus with QEMU/KVM

Introduction

In this article I want to show you how to start with a bare Cisco NX-OS QEMU image guest and configure it to be reachable via the network of the host system. In this approach I am going to use an ‘expect’ script that is going through the initial configuration dialogues of the console of NX-OS.

After having a readily set up system, you can continue with other expect-scripts or configure your Cisco switch with Ansible or the automation of your choice.

The approach is more raw than the approach with Vagrant described in an earlier article. This way works around all SSH and SCP problems that arise when using Vagrant. Besides that, you are not needing VirtualBox virtualization. You can save and load QEMU snapshots using the so-called QEMU monitor.

In the following I want to lead you through the steps to set up your first VM.

The steps are:

  1. Installing the Linux host dependencies.
  2. Downloading the Cisco NX-OS image.
  3. Downloading the wrapper scripts.
  4. Starting the NX-OS VM.
  5. Starting the expect script.
  6. Waiting for the expect script to set up the switch.
  7. Using the NX-OS switch via the exposed network ports.

Linux packages to install

You need to have the packages for QEMU in your Linux system. The following gives the Debian packages needed:

# apt-get install qemu-system-x86 qemu-utils ovmf expect netcat

I want to summarize for every package what it’s for:

  • qemu-system-x86: The x86 QEMU runtime.
  • qemu-utils: Some utils like ‚qemu-img‘ which comes handy when creating a snapshot copy from another VM image.
  • ovmf: An UEFI-enabled firmware for VMs that is needed to boot NX-OS properly.
  • expect: A tool for automating interactive dialogue programs. In our case we’re automating the console-based setup of NX-OS with it.

Downloading a Cisco NX-OS image

You need a Cisco NX-OS image from Cisco. As a precondition you need a valid login and password and license at Cisco.

The images are usually called „Cisco Nexus 9000/3000 Virtual Switch for KVM“. A link for Nexus 9000v and release 9.3(10) is here:

https://software.cisco.com/download/home/286312239/type/282088129/release/9.3(10)

The downloaded file will have a name like nexus9300v.9.3.10.qcow2.

Downloading the wrapper scripts

In this github repo, there are multiple wrapper scripts showing the use of QEMU and also an expect script that is automating the initial setup of a NX-OS switch.

The following is an overview of the relevant files:

  • qemu-snapshot-and-start.sh: Creates a snapshot from a (unaltered) Cisco Nexus image and starts from this snapshot. All data written is written to this snapshot.
  • qemu-start-from-snapshot.sh: Starts a QEMU VM from a previously created snapshot (see above).
  • expect.ex: An expect script that connects to an existing QEMU VM via Console TCP port and goes through the installation

Starting the NX-OS VM

The qemu-snapshot-and-start.sh script requires you to pass the downloaded NX-OS image as an environment varialbe called $SOURCE_IMAGE.

When you have done so, you can call the script.

The steps will then be the following:

Creating an image

Create a copy-on-write image of the input image:

qemu-img create -f qcow2 -b ${SOURCE_IMAGE} ${SNAPSSHOT_IMAGE}

The reason for this is to not alter the ‚clean‘ original image. Besides that, images created using this approach are copy-on-write images, meaning that you don’t need to duplicate the whole VM image. The backing image is required for the VM to work.

Spawning QEMU

The following is the command line the script uses to spawn QEMU.

I want to give you the command and then discuss every option:

qemu-system-x86_64 -smp 2 -enable-kvm \
   -nographic \
   -bios /usr/share/qemu/OVMF.fd \
   -m 8192 \
   -serial telnet:localhost:2023,server=on,wait=off \
   -device ahci,id=ahci0,bus=pci.0 -drive file=${SNAPSSHOT_IMAGE},if=none,id=drive-sata-disk0,id=drive-sata-disk0,format=qcow2 \
   -device ide-hd,bus=ahci0.0,drive=drive-sata-disk0,id=drive-sata-disk0,bootindex=1 \
   -netdev user,id=net0,hostfwd=tcp::3022-:22,hostfwd=tcp::3080-:80,hostfwd=tcp::3443-:443 \
   -device e1000,netdev=net0,mac=aa:bb:cc:dd:ee:ff,multifunction=on,romfile=,bootindex=3

Now let’s discuss every option:

OptionMeaning
-smp 2Have 2 CPU cores.
-enable-kvmEnable KVM virtualization (acceleration).
-nographicDo not show the VGA console of the switch. NX-OS will to serial console in the boot process very early, so the VGA console will be useless very early.
-bios /usr/share/qemu/OVMF.fdSpecify the OVMF bios that is needed by NX-OS to boot.
-m 8192Start with 8 GB of RAM.
-serial telnet:localhost:2023,server=on,wait=offBind the console UART to the hosts TCP port 2023.
-device ahci,id=ahci0,bus=pci.0Register a SATA device in AHCI mode
-drive file=${SNAPSSHOT_IMAGE},if=none,id=drive-sata-disk0,id=drive-sata-disk0,format=qcow2Register the first hard disk to the image
-device ide-hd,bus=ahci0.0,drive=drive-sata-disk0,id=drive-sata-disk0,bootindex=1Bind the previously registered HDD to the first IDE
-netdev user,id=net0, hostfwd=tcp::3022-:22, hostfwd=tcp::3080-:80, hostfwd=tcp::3443-:443Register net0 with multiple TCP sockets from host to guest
-device e1000,netdev=net0,
mac=aa:bb:cc:dd:ee:ff,
multifunction=on,romfile=,bootindex=3
Bind the net0 netdev to a e1000 NIC. Lower the bootindex so OVMF is not trying netboot.
QEMU options explained

The above command will map the following TCP ports from the host to the guest:

HostGuestMeaning
2023consoleThe serial console port
302222SSH port
308080HTTP port (unencrypted, for tcpdump)
3443443HTTPS port (encrypted)
Host to guest mapping in the example above

QEMU commands

Useful QEMU commands are:

  • quit: Quit the VM.
  • savevm $NAME: Save the VM state (HDD + MEM + CPU) to a snapshot called $NAME in the HDD image.
  • loadvm $NAME: Load the VM state from a snapshot called $NAME from the HDD image.

Automating QEMU

QEMU has a console to modify the running VM.

By adding a monitor TCP port, you can access the QEMU console via TCP and send QEMU commands to it (see above). This can be done by adding this to the QEMU command line:

-monitor telnet::45454,server,nowait

Starting the expect script

When QEMU is running and accepting TCP connections to the console port, a bare NX-OS switch will boot and wait for configuration inputs.

This is the point of time where the expect script kicks in.

$ ./expect.ex
....
switch(config)# int mgmt0
int mgmt0
switch(config-if)# ip address dhcp
ip address dhcp
Warning: Disable ip dhcp relay if enabled.
switch(config-if)# exit
exit
switch(config)# copy running-config startup-config
copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete.
switch(config)# exit
exit
switch# 
12:44:28 INFO OK: Expect script finished with success. Expect script took 255 seconds.

It connects to the NX-OS VM using the console port. It does then the following:

  1. Specify boot image if stuck in boot loader (image name may need to be adjusted).
  2. Abort Power On Auto Provisionig.
  3. Deny to enfore secure passwords.
  4. Enter user/password of the ‚admin‘ user.
  5. Not enter the basic config dialog.
  6. Login with the previously defined admin user/password.
  7. Configure the boot image.
  8. Activate DHCP on interface mgmt0 so the QEMU DHCP service can assign an IP address to the switch.
  9. Persist all changes so far to the startup-config.

The whole procedure will take around 4.5 minutes. Because of this it really pays off to take a snapshot of the NX-OS VM once you are done so you can restore the snapshot in seconds.

The expect script is using green-colored lines for informational messages and red-colored lines for error messages.

When the expect script is finished, it will print

OK: Expect script finished with success. Expect script took ... seconds.

Using the NX-OS switch via the exposed network ports

The NX-OS switch is now booted and reachable from the hosts network.

You can log in with a normal login user or an admin user.

UsernamePassword
adminadmin
Users created by the expect script

You can connect to the switch either using telnet and a console port, or SSH:

$ ssh -oPubKeyAuthentication=false -p3022 admin@localhost
ssh -oPubKeyAuthentication=false -p3022 admin@localhost
The authenticity of host '[localhost]:3022 ([127.0.0.1]:3022)' can't be established.
RSA key fingerprint is SHA256:FronHu+9JxcRpCOvGcwpKEF+MZdfWoS94cM6MhcGghg.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[localhost]:3022' (RSA) to the list of known hosts.
User Access Verification
Password: 

Bad terminal type: "xterm-256color". Will assume vt100.
Cisco NX-OS Software
Copyright (c) 2002-2020, Cisco Systems, Inc. All rights reserved.
Nexus 9000v software ("Nexus 9000v Software") and related documentation,
files or other reference materials ("Documentation") are
the proprietary property and confidential information of Cisco
Systems, Inc. ("Cisco") and are protected, without limitation,
pursuant to United States and International copyright and trademark
laws in the applicable jurisdiction which provide civil and criminal
penalties for copying or distribution without Cisco's authorization.

Any use or disclosure, in whole or in part, of the Nexus 9000v Software
or Documentation to any third party for any purposes is expressly
prohibited except as otherwise authorized by Cisco in writing.
The copyrights to certain works contained herein are owned by other
third parties and are used and distributed under license. Some parts
of this software may be covered under the GNU Public License or the
GNU Lesser General Public License. A copy of each such license is
available at
http://www.gnu.org/licenses/gpl.html and
http://www.gnu.org/licenses/lgpl.html
***************************************************************************
*  Nexus 9000v is strictly limited to use for evaluation, demonstration   *
*  and NX-OS education. Any use or disclosure, in whole or in part of     *
*  the Nexus 9000v Software or Documentation to any third party for any   *
*  purposes is expressly prohibited except as otherwise authorized by     *
*  Cisco in writing.                                                      *
***************************************************************************
switch# 

Summary

In this article I’ve shown how to use QEMU with KVM.

VM snapshots are stored in the QCOW2 image file and can be restored later on per command towards the QEMU monitor port. This saves a lot of time when in a testing environment.

The approach feels raw from the first view, but gives you a lot of control over your VM.

Compared to the vagrant approach discussed in an article before, this approach spawns less processes, does not mess around with the switches default shells and has more control over the image files.

Network Device Simulation for Cisco Nexus with Vagrant

Introduction

This is a step-by-step instruction on how to get started with virtualization with Vagrant, VirtualBox and Cisco NX-OS images. This comes handy when developing network automation approaches and testing them in a test environment with no cost of real hardware.

An article showing a more raw QEMU/KVM based approach can be found here.

Vagrant is a virtualization and automation frontend for multiple virtualization backends like virtualbox, libvirt, vmware. It comes with a image management and scripting approach called ‚Vagrantfile‘.

Please note that for most vagrant operations to work you need to have the Vagrantfile in your current working directory.

Cisco provides Vagrant images of their Nexus 9300/9500 virtual switch series.

In summary, you are needing these steps:

  1. Install OS dependencies
  2. Download Cisco NX-OS box image
  3. Import Cisco NX-OS box image
  4. Write Vagrantfile
  5. Start VM with vagrant up
  6. If needed, do provisioning with vagrant provision
  7. Stop VM with vagrant destroy -f

Setup

Linux packages to install

You need to have the packages for vagrant and virtualbox in your Linux system. The following gives the Debian packages needed:

# apt-get install vagrant virtualbox

Downloading a Cisco NX-OS image

You need a Cisco NX-OS image from Cisco. As a precondition you need a valid login and password and license at Cisco.

The images are usually called „Cisco Nexus 9000/3000 Virtual Switch for Vagrant“. A link for Nexus 9000v and release 9.3(10) is here:

https://software.cisco.com/download/home/286312239/type/282088129/release/9.3(10)

The downloaded file will have a name like nexus9300v.9.3.10.box.

Importing the box to Vagrant

To import the downloaded box-image into Vagrant, issue the command

$ vagrant box add nexus9300v.9.3.5 ~/Downloads/nexus9300v.9.3.10.box –force

This will import the file towards the logical name nxos/9.3.10. This is the name of the „config.vm.box“ you’ll need later in the Vagrantfile.

Afterwards you can see the box in your local repository:

$ vagrant box list
debian/bullseye64 (virtualbox, 11.20220718.1)
nexus9300v.9.3.10 (virtualbox, 0)
nexus9300v.9.3.5  (virtualbox, 0)
ubuntu/trusty64   (virtualbox, 20190514.0.0)

Creating a Vagrantfile for the box

Create a new text file called Vagrantfile defining your setup. This defines your whole VM setup and how to configure your NOS.

The following is a table of mapped TCP/UDP ports of the Vagrant file shown in the next section:

ServiceProtoGuestHostDescription
ConsoleTCPUART12023Console port, not directly assigned in Vagrantfile
SSHTCP223022SSH service, also used for vagrant provisioning
HTTPTCP803080HTTP service, not used anymore by current NX-OS
HTTPSTCP4433443HTTPS service for „feature nxapi“
NetConfTCP8303830NetConf service
SNMPUDP1613161SNMP UDP port forward
TCP / UDP port forwardings as configured in the Vagrantfile

Vagrantfile

The following is a Vagrantfile with the guest TCP/UDP ports mentioned above mapped to host ports. A Vagrantfile is also given on this Github repo.

# -*- mode: ruby -*-
# vi: set ft=ruby :
 
Vagrant.configure("2") do |config|
 
  # name of the box as in "vagrant box add"
  config.vm.box = "nexus9300v.9.3.5"
 
  # foward ssh port
  config.vm.network :forwarded_port, guest: 22, host: 3022, id: 'ssh', auto_correct: true
  # foward http port not needed anymore, current NXOS uses TCP 443 for feature nxapi
  #config.vm.network :forwarded_port, guest: 80, host: 3080, id: 'http', auto_correct: true
  # enable "feature nxapi", browse to "https://localhost:3443/"
  config.vm.network :forwarded_port, guest: 443, host: 3443, id: 'https', auto_correct: true
  # netconf service
  config.vm.network :forwarded_port, guest: 830, host: 3830, id: 'netconf', auto_correct: true
  # forward UDP snmp port
  config.vm.network :forwarded_port, guest: 161, host: 3161, protocol: 'udp', id: 'snmp', auto_correct: true
  config.ssh.shell = "run bash"
  config.vm.provider "virtualbox" do |vb|
#     vb.gui = true
     vb.cpus = "2"
     vb.memory = "8192"
     vb.customize ["modifyvm", :id, "--firmware", "efi"]
     # telnet console, usually on tcp 2023
     vb.customize ["modifyvm", :id, "--uart1", "0x3F8", "4"]
    end
end

Starting up your VM

Change the working directory towards your Vagrantfile definition. 

$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'nexus9300v.9.3.5'...
==> default: Matching MAC address for NAT networking...
==> default: Setting the name of the VM: nexus_default_1661767270999_61318
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
    default: Adapter 2: intnet
==> default: Forwarding ports...
    default: 22 (guest) => 3022 (host) (adapter 1)
    default: 80 (guest) => 3080 (host) (adapter 1)
    default: 443 (guest) => 3443 (host) (adapter 1)
    default: 830 (guest) => 3830 (host) (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:3022
    default: SSH username: vagrant
    default: SSH auth method: private key

NX-OS is now starting up. After some time the vagrant output will print something about an „insecure key“ (which is standard in vagrant)

    default: Vagrant insecure key detected. Vagrant will automatically replace
    default: this with a newly generated keypair for better security.
    default:
    default: Inserting generated public key within guest...
==> default: Attempting graceful shutdown of VM...
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!
 
shutdown -h now
 
Stdout from the command:
 
 
 
Stderr from the command:

Using your VM

You can now start up the VirtualBox user interface and take a look at your VM. Don’t expect too much from the VGA output, NX-OS will print almost nothing there and allow no login via the VGA console. See

Pre-defined users and their passwords

After booting, you can log in to the UART1 console or SSH-forwarded port using the following users that are preconfigured in the NX-OS box:

UserPasswordPurpose
adminadminStandard Cisco admin user.
vagrantvagrantStandard vagrant automation user for SSH. Is used by „vagrant ssh“ (see below).
Users and passwords that are generated by default

Usage via console UART1

You can connect towards the UART1 to access the console and see NX-OS booting:

$ telnet localhost 2023
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
cat: /sys/module/klm_cctrli*/parameters/cctrl_hw_rev_maj*: No such file or directory
cp: cannot stat '/nxos/tmp/vdc_1_sup1': No such file or directory
rm: cannot remove '/isan/bin/bios_imgs/*': No such file or directory
rm: cannot remove '/isan/lib/libtahusdcli.so': No such file or directory
 
Linking n9k flash devices
creating flash devices BOOT_DEV= sda
mknod: '/dev/sda7': File exists
Replacing inittab for N9Kv pEoR node - vsup
INIT: version 2.88 booting
Unsquashing rootfs ...
Total size needed in bootflash is 108588
check bootflash : OK
Total size needed in bootflash is 41032
check bootflash : OK
No TOR virtual chassis
Enabling 8250 serial driver spurious INTs workaround
Installing isan procfs isan_proc_name=isan26... done.
...

Usage via „vagrant ssh“

Vagrant has a pre-defined wiring towards the VMs ssh port for automation purposes. You can access the ssh port after VM boot using the vagrant command

$ vagrant ssh
 
Cisco NX-OS Software
Copyright (c) 2002-2020, Cisco Systems, Inc. All rights reserved.
Nexus 9000v software ("Nexus 9000v Software") and related documentation,
files or other reference materials ("Documentation") are
the proprietary property and confidential information of Cisco
Systems, Inc. ("Cisco") and are protected, without limitation,
pursuant to United States and International copyright and trademark
laws in the applicable jurisdiction which provide civil and criminal
penalties for copying or distribution without Cisco's authorization.
 
Any use or disclosure, in whole or in part, of the Nexus 9000v Software
or Documentation to any third party for any purposes is expressly
prohibited except as otherwise authorized by Cisco in writing.
The copyrights to certain works contained herein are owned by other
third parties and are used and distributed under license. Some parts
of this software may be covered under the GNU Public License or the
GNU Lesser General Public License. A copy of each such license is
available at
http://www.gnu.org/licenses/gpl.html and
http://www.gnu.org/licenses/lgpl.html
***************************************************************************
*  Nexus 9000v is strictly limited to use for evaluation, demonstration   *
*  and NX-OS education. Any use or disclosure, in whole or in part of     *
*  the Nexus 9000v Software or Documentation to any third party for any   *
*  purposes is expressly prohibited except as otherwise authorized by     *
*  Cisco in writing.                                                      *
***************************************************************************
Nexus9000v#

Provisioning your VM

Usually Vagrant will execute the provisioning parts of your VM. For NX-OS, there are usually some problems with the vagrant startup. You usually need to call „vagrant provision“ by hand if you have provisioning parts in your Vagrantfile. Here’s an example of a Ansible provisioning:

$ vagrant provision
==> nxos1: Running provisioner: ansible...
    nxos1: Running ansible-playbook...
 
PLAY [Provision NX-OS Devices] *************************************************
 
TASK [Pause to complete boot] **************************************************
[WARNING]: ansible-pylibssh not installed, falling back to paramiko
Pausing for 5 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)

Stopping your VM

The VM can be stopped in multiple ways. The safest way is to destroy the VM:

$ vagrant destroy -f
==> default: Forcing shutdown of VM...
==> default: Destroying VM and associated drives...

Snapshotting the VM state

Snapshotting the state of the VM and restoring it is a way to quickly roll back the VM to a defined state.

This only works while the VM is running. As soon as the VM is destroyed, also the snapshots go away.

Saving a snapshot

The following example saves a snapshot called „aftersnap“:

$ vagrant snapshot save default aftersnap
==> default: Snapshotting the machine as 'aftersnap'...
==> default: Snapshot saved! You can restore the snapshot at any time by
==> default: using `vagrant snapshot restore`. You can delete it using
==> default: `vagrant snapshot delete`.

Listing the snapshots

The following lists all snapshots. In this example there were two snapshots saved:

$ vagrant snapshot list
==> default:
aftersnap
init

Restoring a snapshot

The following restores a snapshot from disk and replaces the current VM state with the snapshotted state

$ vagrant snapshot restore default aftersnap2
==> default: Forcing shutdown of VM...
==> default: Restoring the snapshot 'aftersnap2'...
==> default: Resuming suspended VM...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:3022
    default: SSH username: vagrant
    default: SSH auth method: private key
The configured shell (config.ssh.shell) is invalid and unable
to properly execute commands. The most common cause for this is
using a shell that is unavailable on the system. Please verify
you're using the full path to the shell and that the shell is
executable by the SSH user.

FAQ

Virtualbox can’t be installed

You might need to add another repository to your Debian system. See here.

Isn’t VirtualBox having licensing issues?

VirtualBox itself is licensed under GPL v2. VirtualBox extension pack is under a Oracle license, but not needed for Vagrant.

What’s the boot time for a NX-OS switch?

The boot time is about 4:30 minutes on my Dell Latitude 5501.

Can the boot time be reduced?

One way to not have the boot time for every test case, I recommend keeping the VM running and just restoring VM snapshots.

VM snapshot restore costs only about 40s in contrast to 270s full boot (6 times faster).

What are the memory requirements?

NX-OS needs at least 6 GB of RAM according to many internet posts, ideally it requires 8 GB RAM. This means you need a lot of RAM.

What’s already configured in the VM?

The Vagrant box from Cisco has already some things pre-configured for the Vagrant setup to work:

  • User vagrant for „vagrant ssh“
  • Interface „mgmt0“ is up and has an IP address from VirtualBox DHCP?

Why does shell provisioning not work?

Vagrant shell provisioning does not work because of several reasons:

  • Secure copy: Access Denied (Net::SCP::Error)
    • Problem: Vagrant is using a scp-based mechanism and scp is not enabled per default.
    • Solution: „feature scp“ needs to be enabled.
  • scp: /bootflash/tmp/vagrant-shell: No such file or directory (Net::SCP::Error)
    • Problem: The filesystem path /bootflash/tmp is not existing.
    • Solution: You can override vagrants path with the ‚upload_path‘ option.

config.vm.provision "shell", upload_path: "tmpfile", inline: "echo bash"

  • The SSH command responded with a non-zero exit status
    • Problem: Changing the uploaded file attributes fails because vagrant is logging in with a Cisco CLI, not a bash

INFO ssh: Execute: chmod +x 'tmpfile' && tmpfile (sudo=true)
DEBUG ssh: Exit status: 20

  • Solution: Unknown

The topic can be troubleshooted using

$ vagrant provision --debug

Cisco proposes to use Ansible provisioning which is more complex.

Is there a demo on how to do Ansible provisioning?

Yes, there is a sample repo here.

Is there a way to do a simple shell provisioning?

You can add a simple sequence like this to your Vagrantfile:

if VAGRANT_COMMAND = "up"
  config.ssh.shell = "conf t ; hostname helloworld ; copy r s"
end

Is it possible to simulate multiple switches / bigger topology?

There are Vagrantfile examples defining bigger topologies, but I did not test them. Sample: https://github.com/hpreston/vagrant_net_prog/blob/master/nx-os/two-switches/Vagrantfile

Can SNMP be also simulated?

Yes, you only need to configure SNMP on your switch and specify the fowarded UDP port number in the command line (here 3161)

$ snmpwalk -v2c -cpublic localhost:3161

Summary

In this article the basics of Vagrant virtualization of a Cisco NX-OS switch is shown.

There are some challenges when it comes to Vagrant operations that require a full-blown Linux system, but find a network device. These are SSH-based automation and SCP. I found automation to not work reliably with simple Vagrant provision commands, but to work with Ansible scripting.

NX-OS startup time is slow for testing environments that require a clean test target. Vagrant offers a snapshot mechanism to reset your VM towards a named well-defined state. The snapshot mechanism of Vagrant is heavily dependant on the virtualization solution it uses. The location of your snapshot file(s) is defined by VirtualBox.

With Ansible scripting provisioning, automation worked pretty well.

Further reading

In the following I want to give you pointers to further information: