How to install Linux VM on FreeBSD using bhyve and ZFS

How to install Linux VM on FreeBSD using bhyve

Our sample set up is as follows:

Our network interface name is em0 with static IP. We will create a network bridge called em0bridge, which will act as a logical link between two or more networks that use the same framing format. We will create a new tap interface for each VM and attach it to our VM. Next, that tap interface will be added to em0bridge. Creating a virtual machine in bhyve is configuring the FreeBSD host system involves various steps. Enough of theory. Let us get our hands dirty with actual commands.

Step 1 – Make tap device UP by default

Append the following entry to the /etc/sysctl.conf:
echo "net.link.tap.up_on_open=1" >> /etc/sysctl.conf
sysctl net.link.tap.up_on_open=1
When net.link.tap.up_on_open is 1/non-zero value, the tunnel device will be marked “up” when the control device is opened.

Step 2 – Loading vmm FreeBSD kernel bhyve virtual machine monitor

Use the kldload command to load FreeBSD kernel module at runtime, run:
kldload vmm
kldload nmdm
Make sure we load the driver as a module at boot time:

echo 'vmm_load="YES"' >> /boot/loader.conf
echo 'nmdm_load="YES"' >> /boot/loader.conf
echo 'if_tap_load="YES"' >> /boot/loader.conf
echo 'if_bridge_load="YES"' >> /boot/loader.conf

Step 3 – Creating bridge and tap device

Create a new bridge, run:
ifconfig bridge create
Add the interface named em0 as a member of the bridge. The interface is put into promiscuous mode so that it can receive every packet sent on the network:
ifconfig bridge0 addm em0
Set the interface name to em0bridge for ease of identification:
ifconfig bridge0 name em0bridge
ifconfig em0bridge up
Finally create a new tap interface called tap0 and add it to the em0bridge:
ifconfig tap0 create
ifconfig em0bridge addm tap0
Make sure we set and activate network devices to auto boot in /etc/rc.conf file:

# byhve and jail settings bridges 
cloned_interfaces="bridge0 tap0"
ifconfig_bridge0_name="em0bridge"
ifconfig_em0bridge="addm em0 addm tap0 up"

Step 4 – Creating ZFS volumes for Linux bhyve guest/VM

ZFS gives us significant performance over disk image. We can easily do other operations, such as cloning and backups. Hence, I am going to create a 20GB volume as follows:
zfs create -V20G -o volmode=dev zroot/debianvm

Step 5 – Downloading Debian Linux installer iso image

Use the wget command to download image. Alternatively, we can use curl to download image too:
cd /tmp/
wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-10.7.0-amd64-netinst.iso

Step 6 – Install Grub support for bhyve under FreeBSD

GNU/GRUB is a multiboot boot loader and is recommended for Linux guests. The following package allows us to run the grub-bhyve binary. In other words, it is allowing us booting of non-FreeBSD guest operating systems such as Debian or Ubuntu Linux.
pkg install grub2-bhyve

Step 7 – Installing Linux VM on FreeBSD

Now everything is in place. It is time to install the actual Debian Linux 10.x VM under FreeBSD using the bhyve command:
bhyve -c 1 -m 1G -w -H \
-s 0,hostbridge \
-s 3,ahci-cd,/tmp/debian-10.7.0-amd64-netinst.iso \
-s 4,virtio-blk,/dev/zvol/zroot/debianvm \
-s 5,virtio-net,tap0 \
-s 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600,wait \
-s 30,xhci,tablet \
-s 31,lpc -l com1,stdio \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
debianvm

Understanding bhyve command-line options:

• -c : number of virtual cpus
• -H : host filesystem to export to the loader
• -l : the OS loader to use (uefi needed for non-FreeBSD oses such as Linux/Windows guests)
• -m : ram/memory size for VM
• -w : ignore unimplemented MSRs
• -s : Configure a virtual PCI slot and other function such as hard disk, cdrom, and other devices

Getting help is easy too:
man bhyve
bhyve -s help
bhyve -l help

A note about VNC client:

To access Debian 10 Linux installer, open your VNC client and type server_IP:5900. For instance:

Using VNC client to access Debian 10 installer from my Ubuntu Linux desktop

Debian 10 Linux VM installer running on FreeBSD bhyve host

Step 8 – Booting Debian 10 Linux bhvye VM on FreeBSD

After installing a Debian 10 virtual machine, we rebooted the virtual machine and forced bhyve to exit. Therefore, the instance of the virtual machine needs to be destroyed before it can be started again:
bhyvectl --destroy --vm=debianvm
Finally, boot the Debian 10 virtual machine and enjoy the fruits of our labor:
bhyve -c 2 -m 1G -w -H \
-s 0,hostbridge \
-s 4,virtio-blk,/dev/zvol/zroot/debianvm \
-s 5,virtio-net,tap0 \
-s 29,fbuf,tcp=0.0.0.0:5900,w=1024,h=768,wait \
-s 30,xhci,tablet \
-s 31,lpc -l com1,stdio \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
debianvm

Debian 10 booted successfully under FreeBSD bhyve without UEFI boot problems

Step 9 – Starting Bhyve Debian 10 VM on boot

Create a simple shell script as follows:

#!/bin/sh
# Name: startdebianvm
# Purpose: Simple script to start my Debian 10 VM using bhyve on FreeBSD
# Author: Vivek Gite {https://www.cyberciti.biz} under GPL v2.x+
-------------------------------------------------------------------------
# Lazy failsafe (not needed but I will leave them here)
ifconfig tap0 create
ifconfig em0bridge addm tap0
if ! kldstat | grep -w vmm.ko 
then
	kldload -v vmm
fi
if ! kldstat | grep -w nmdm.ko
then
	kldload -v nmdm
fi
bhyve -c 1 -m 1G -w -H \
-s 0,hostbridge \
-s 4,virtio-blk,/dev/zvol/zroot/debianvm \
-s 5,virtio-net,tap0 \
-s 29,fbuf,tcp=0.0.0.0:5900,w=1024,h=768 \
-s 30,xhci,tablet \
-s 31,lpc -l com1,stdio \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
debianvm

Run it using cron job. We can execute cron job after system reboot:
@reboot /path/to/startdebianvm

Summing up

That is all for now. I hope you enjoyed this tutorial. Next time we will talk about firewall NAT settings, reverse proxy, and cloud images. FreeBSD ports have many Bhyve management tools too. Make sure you check out man page and handbook chapter too:
man 8 bhyve