How to install Linux VM on FreeBSD using bhyve and ZFS

Using bhyve on FreeBSD and installing Debian Linux 10

Bhyve (“BSD hypervisor”) is a free and opensource hypervisor for FreeBSD. We can use Bhyve to run Linux, Windows, and *BSD guest operating system as a virtual machine. Let us see how to install Linux VM using Bhyve on FreeBSD host operating system along with ZFS and bridged networking.



How to install Linux VM on FreeBSD using bhyve

Our sample set up is as follows:
How to install Linux VM on FreeBSD using bhyve
Our network interface name is em0 with static IP. We will create a network bridge called em0bridge, which will act as a logical link between two or more networks that use the same framing format. We will create a new tap interface for each VM and attach it to our VM. Next, that tap interface will be added to em0bridge. Creating a virtual machine in bhyve is configuring the FreeBSD host system involves various steps. Enough of theory. Let us get our hands dirty with actual commands.

Step 1 – Make tap device UP by default

Append the following entry to the /etc/sysctl.conf:
echo "net.link.tap.up_on_open=1" >> /etc/sysctl.conf
sysctl net.link.tap.up_on_open=1

When net.link.tap.up_on_open is 1/non-zero value, the tunnel device will be marked “up” when the control device is opened.

Step 2 – Loading vmm FreeBSD kernel bhyve virtual machine monitor

Use the kldload command to load FreeBSD kernel module at runtime, run:
kldload vmm
kldload nmdm

Make sure we load the driver as a module at boot time:

echo 'vmm_load="YES"' >> /boot/loader.conf
echo 'nmdm_load="YES"' >> /boot/loader.conf
echo 'if_tap_load="YES"' >> /boot/loader.conf
echo 'if_bridge_load="YES"' >> /boot/loader.conf

Step 3 – Creating bridge and tap device

Create a new bridge, run:
ifconfig bridge create
Add the interface named em0 as a member of the bridge. The interface is put into promiscuous mode so that it can receive every packet sent on the network:
ifconfig bridge0 addm em0
Set the interface name to em0bridge for ease of identification:
ifconfig bridge0 name em0bridge
ifconfig em0bridge up

Finally create a new tap interface called tap0 and add it to the em0bridge:
ifconfig tap0 create
ifconfig em0bridge addm tap0

Make sure we set and activate network devices to auto boot in /etc/rc.conf file:

# byhve and jail settings bridges 
cloned_interfaces="bridge0 tap0"
ifconfig_bridge0_name="em0bridge"
ifconfig_em0bridge="addm em0 addm tap0 up"

Step 4 – Creating ZFS volumes for Linux bhyve guest/VM

ZFS gives us significant performance over disk image. We can easily do other operations, such as cloning and backups. Hence, I am going to create a 20GB volume as follows:
zfs create -V20G -o volmode=dev zroot/debianvm

Step 5 – Downloading Debian Linux installer iso image

Use the wget command to download image. Alternatively, we can use curl to download image too:
cd /tmp/
wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-10.7.0-amd64-netinst.iso

Step 6 – Install Grub support for bhyve under FreeBSD

GNU/GRUB is a multiboot boot loader and is recommended for Linux guests. The following package allows us to run the grub-bhyve binary. In other words, it is allowing us booting of non-FreeBSD guest operating systems such as Debian or Ubuntu Linux.
pkg install grub2-bhyve

Step 7 – Installing Linux VM on FreeBSD

Now everything is in place. It is time to install the actual Debian Linux 10.x VM under FreeBSD using the bhyve command:
bhyve -c 1 -m 1G -w -H \
-s 0,hostbridge \
-s 3,ahci-cd,/tmp/debian-10.7.0-amd64-netinst.iso \
-s 4,virtio-blk,/dev/zvol/zroot/debianvm \
-s 5,virtio-net,tap0 \
-s 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600,wait \
-s 30,xhci,tablet \
-s 31,lpc -l com1,stdio \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
debianvm

Understanding bhyve command-line options:

  • -c : number of virtual cpus
  • -H : host filesystem to export to the loader
  • -l : the OS loader to use (uefi needed for non-FreeBSD oses such as Linux/Windows guests)
  • -m : ram/memory size for VM
  • -w : ignore unimplemented MSRs
  • -s : Configure a virtual PCI slot and other function such as hard disk, cdrom, and other devices

Getting help is easy too:
man bhyve
bhyve -s help
bhyve -l help

A note about VNC client:

To access Debian 10 Linux installer, open your VNC client and type server_IP:5900. For instance:

Accessing FreeBSD as a Host with bhyve over VNC

Using VNC client to access Debian 10 installer from my Ubuntu Linux desktop

And Voilà, we are greeted with Debian 10 installer:
Using bhyve on FreeBSD and installing Debian Linux 10

Debian 10 Linux VM installer running on FreeBSD bhyve host

Now all you have to do is follow the installer till you reach the Linux reboot screen. We need to make one manual adjustment to Debian 10 VM bootloader under FreeBSD. Otherwise, we will end up non-booting Debian 10 bhvye VM. We don’t want that do we now? Click on the Go Back button:
Fixing GRUB bhyve Debian 10 issue on FreeBSD 1
Make sure you scroll down and choose the “Execute a shell” option and click on the “Continue” button:
Debian 10 Linux not booting on FreeBSD fix 2
Type the following mkdir command and cp command:
mkdir /target/boot/efi/EFI/BOOT/
# copy file - workaround for bhyve grub package #
# Pay attention to destination file bootx64.efi #

cp /target/boot/efi/EFI/debian/grubx64.efi /target/boot/efi/EFI/BOOT/bootx64.efi

And we are done fixing GRUB. Type the “exit” command to return to the installation menu:
exit
Fixing grub issues
Make sure you finish the installation and reboot Debian 10 bhvye VM:
Finish the installation

Step 8 – Booting Debian 10 Linux bhvye VM on FreeBSD

After installing a Debian 10 virtual machine, we rebooted the virtual machine and forced bhyve to exit. Therefore, the instance of the virtual machine needs to be destroyed before it can be started again:
bhyvectl --destroy --vm=debianvm
Finally, boot the Debian 10 virtual machine and enjoy the fruits of our labor:
bhyve -c 2 -m 1G -w -H \
-s 0,hostbridge \
-s 4,virtio-blk,/dev/zvol/zroot/debianvm \
-s 5,virtio-net,tap0 \
-s 29,fbuf,tcp=0.0.0.0:5900,w=1024,h=768,wait \
-s 30,xhci,tablet \
-s 31,lpc -l com1,stdio \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
debianvm

success Debian 10 Linux booting

Debian 10 booted successfully under FreeBSD bhyve without UEFI boot problems

Step 9 – Starting Bhyve Debian 10 VM on boot

Create a simple shell script as follows:

#!/bin/sh
# Name: startdebianvm
# Purpose: Simple script to start my Debian 10 VM using bhyve on FreeBSD
# Author: Vivek Gite {https://www.cyberciti.biz} under GPL v2.x+
-------------------------------------------------------------------------
# Lazy failsafe (not needed but I will leave them here)
ifconfig tap0 create
ifconfig em0bridge addm tap0
if ! kldstat | grep -w vmm.ko 
then
	kldload -v vmm
fi
if ! kldstat | grep -w nmdm.ko
then
	kldload -v nmdm
fi
bhyve -c 1 -m 1G -w -H \
-s 0,hostbridge \
-s 4,virtio-blk,/dev/zvol/zroot/debianvm \
-s 5,virtio-net,tap0 \
-s 29,fbuf,tcp=0.0.0.0:5900,w=1024,h=768 \
-s 30,xhci,tablet \
-s 31,lpc -l com1,stdio \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
debianvm

Run it using cron job. We can execute cron job after system reboot:
@reboot /path/to/startdebianvm

Summing up

That is all for now. I hope you enjoyed this tutorial. Next time we will talk about firewall NAT settings, reverse proxy, and cloud images. FreeBSD ports have many Bhyve management tools too. Make sure you check out man page and handbook chapter too:
man 8 bhyve


🐧 Get the latest tutorials on Linux, Open Source & DevOps via RSS feed or Weekly email newsletter.

🐧 10 comments so far... add one

CategoryList of Unix and Linux commands
Disk space analyzersdf ncdu pydf
File Managementcat tree
FirewallAlpine Awall CentOS 8 OpenSUSE RHEL 8 Ubuntu 16.04 Ubuntu 18.04 Ubuntu 20.04
Network UtilitiesNetHogs dig host ip nmap
OpenVPNCentOS 7 CentOS 8 Debian 10 Debian 8/9 Ubuntu 18.04 Ubuntu 20.04
Package Managerapk apt
Processes Managementbg chroot cron disown fg jobs killall kill pidof pstree pwdx time
Searchinggrep whereis which
User Informationgroups id lastcomm last lid/libuser-lid logname members users whoami who w
WireGuard VPNAlpine CentOS 8 Debian 10 Firewall Ubuntu 20.04
10 comments… add one
  • SKull Jan 28, 2021 @ 19:01

    Good read! But you shouldn’t use echo to change base system config files. Use sysrc instead, it provides a minimal form of syntax checking.

    • 🐧 Vivek Gite Jan 29, 2021 @ 3:20

      ah, yes. will update it later. thank you for the feedback.

  • Jupi Jan 31, 2021 @ 20:48

    Any idea how to get pulseaudio listening in tcp on host and connecting to it from vm?

  • Fredo Feb 6, 2021 @ 21:38

    Little typo on Step 8 – Booting Deiban 10

  • Mats Mar 5, 2021 @ 20:00

    Great howto!
    Thanx!

  • otis Mar 6, 2021 @ 11:11

    There is a good example of rc script in /usr/share/examples/bhyve.

  • Brandin May 10, 2021 @ 23:15

    Does FreeBSD support vfio / pcie passthrough for hardware acceleration on the virtual machine guest?

Leave a Reply

Your email address will not be published.

Use HTML <pre>...</pre> for code samples. Still have questions? Post it on our forum