How to set up FreeBSD 12 VNET jail with ZFS

What is a VNET on FreeBSD 12?

VNET is a network subsystem virtualization infrastructure for FreeBSD. We can use VNET to run a firewall or VPN server insider jail in an isolated environment. In other words, we create the FreeBSD jail with its virtual network stack, with its network interfaces, addresses, routing table, and so on. The FreeBSD 12 kernel has the VIMAGE option by default. On FreeBSD 11.x and earlier, we compile the kernel to add VIMAGE support. See how to set up jails on FreeBSD 11.x with VNET for more info.

How to set up FreeBSD 12 VNET jail with ZFS

The procedure for setting up FreeBSD 12 VNET jail with ZFS is as follows. I am using FreeBSD 12.2 release with patch level 2. Verify FreeBSD version:
$ freebsd-version

Step 1 – Creating a new zfs data set for FreeBSD jails

You need to set up the filesystem. Use the zpool command to get a list of configured zfs:
# zpool list
Let us run the following zfs command for zroot mounted at /jails/:
# zfs create -o mountpoint=/jails zroot/jails
Next create a zfs data set called fullbasejail, enter:
# zfs create zroot/jails/fullbasejail
Verify it:
# zfs list
Please note down the /jails/fullbasejail.

Step 2 – Configure the base FreeBSD 12 jail

Let us set up a new base jail template for our work. The syntax is as follows for preparing jail using bsdinstall command:
# export BSDINSTALL_DISTSITE=https://download.freebsd.org/ftp/releases/amd64/12.2-RELEASE/
# bsdinstall jail /jails/fullbasejail

Apply any updates and compare the FreeBSD against a “known good” index of the installed release:
# freebsd-update -b /jails/fullbasejail fetch install
# freebsd-update -b /jails/fullbasejail IDS
Here is what we see. I will ignore /etc/* file warning as I set up root password for my base jail:

src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 3 mirrors found.
Fetching metadata signature for 12.2-RELEASE from update4.freebsd.org... done.
Fetching metadata index... done.
Inspecting system... done.
/etc/master.passwd has SHA256 hash 070a37fe90f5e2fce8191ac77eeb416e3a6b7ab6076fa22badef1f63d4657c01, but should have SHA256 hash 1a7dee96fb231be3e28b5fc2489dcffe61632ab1f6406e4e53f6b149de1e923b.
/etc/pwd.db has SHA256 hash b4172a427ad0f8a2b5a989de5eba8258682818226bdd354dda2c58efc4861daf, but should have SHA256 hash 8535fd83b62622bd4aa9833e2691aa12c65e3baebbf00f018092fb4498891b12.
/etc/spwd.db has SHA256 hash b97d48eebe549442695c40431e9728065606e31cafa87efbe9e655423ca11fff, but should have SHA256 hash ed987aebc759f2aef299909834d82caa8e2ff373bd762a82eeb89b43eafc4b59.

Step 3 – Creating a new FreeBSD 12 jail from the base jail

Alright, our base jail for quick deployment is ready. We can clone the base jail to create a new jail:
# zfs snapshot zroot/jails/fullbasejail@12.2-RELEASE-p2
# List snapshots:
# zfs list -t snapshot
Finally create a new jail called ‘demojail’ from the zfs snapshot we created eariler:
# zfs send -R zroot/jails/fullbasejail@12.2-RELEASE-p2 | zfs receive zroot/jails/demojail
We can also do a clone instead of send/receive. A clone is a writable volume or file system whose initial contents are the same as the dataset from which it was created. As with snapshots, creating a clone is nearly instantaneous and initially consumes no additional disk space. Besides, you can snapshot a clone. For example:
# zfs clone zroot/jails/fullbasejail@12.2-RELEASE-p2 zroot/jails/demojail
We can now easily create multiple jails such as mail, www, db and so on:
# zfs send -R zroot/jails/fullbasejail@12.2-RELEASE-p2 | zfs receive zroot/jails/mail
# zfs send -R zroot/jails/fullbasejail@12.2-RELEASE-p2 | zfs receive zroot/jails/www
# zfs send -R zroot/jails/fullbasejail@12.2-RELEASE-p2 | zfs receive zroot/jails/db
## CLONE ##
# zfs clone zroot/jails/fullbasejail@12.2-RELEASE-p2 zroot/jails/vpn


Step 4 – Configuring basic jail stuff

We need to make sure our jail has the right timezone, dns setting, a hostname, ip address and other stuff in rc.conf. Please note that you need to be in /jails/demojail/ directory. Make sure you adjust paths according to your BSD setup:
# cd /jails/demojail
Set up a dns by creating a file named etc/resolv.conf inside /jails/demojail/:
# vi etc/resolv.conf
Update/edit/append the following (set dns nameserver as per your setup):
nameserver 192.168.2.254
Save and close the file in vim/vi text editor. Set up timezone for your FreeBSD jail using the ln command (I am setting up to IST):
# ln -v usr/share/zoneinfo/Asia/Kolkata etc/localtime
Next you need to setup rc.conf inside /jails/demojail/:
# vi etc/rc.conf
My settings:

# jail hostname #
host_hostname="demojail"
 
# IP address and routing #
# e0b_demojail #
ifconfig_e0b_demojail="inet 192.168.2.250 netmask 255.255.255.0"
defaultrouter="192.168.2.254"
 
# Start or stop services #
cron_flags="$cron_flags -J 15"
sendmail_enable="NONE"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
syslogd_flags="-c -ss"
ipv6_activate_all_interfaces="NO"
sshd_enable="NO"

Save and close the file. Create or update the etc/periodic.conf file inside /jails/demojail/ too:
# vi etc/periodic.conf

## Disable the predefined scheduled jobs which are not required for my jails ##
daily_show_success="NO"
weekly_show_success="NO"
monthly_show_success="NO"
security_show_success="NO"
 
daily_output="/var/log/daily.log"
daily_status_security_output="/var/log/daily.log"
weekly_output="/var/log/weekly.log"
weekly_status_security_output="/var/log/weekly.log"
monthly_output="/var/log/monthly.log"
monthly_status_security_output="/var/log/monthly.log"
 
daily_clean_hoststat_enable="NO"
daily_status_mail_rejects_enable="NO"
daily_status_mailq_enable="NO"
daily_queuerun_enable="NO"
 
daily_status_disks_enable="NO"
daily_status_zfs_zpool_list_enable="NO"
daily_status_network_enable="NO"
daily_status_uptime_enable="NO"
daily_ntpd_leapfile_enable="NO"
weekly_locate_enable="NO"
weekly_whatis_enable="NO"
security_status_chksetuid_enable="NO"
security_status_neggrpperm_enable="NO"
security_status_chkuid0_enable="NO"
security_status_ipfwdenied_enable="NO"
security_status_ipfdenied_enable="NO"
security_status_ipfwlimit_enable="NO"
security_status_ipf6denied_enable="NO"
security_status_tcpwrap_enable="NO"

See /etc/defaults/periodic.conf and man page for more info on those settings:
man 5 periodic.conf

Step 5 – Jail config via jail.conf on the host

Edit or create the /etc/jail.conf as follows:
# vi /etc/jail.conf
Sample config:

demojail {
        host.hostname = "demojail";   # hostname
        path = "/jails/demojail";     # root directory
        exec.clean;
        exec.system_user = "root";
        exec.jail_user = "root";
        # ##########################################################################
        # netgraph/vnet config info
        # e0b is my vnet
        # em0 is my physical network interface connected to the LAN (use ifconfig)
        # jib is located in /usr/local/sbin
        # demojail is my jail name
        # ##########################################################################
        vnet;
        vnet.interface = "e0b_demojail";               # vnet interface(s)
        exec.prestart += "jib addm demojail em0";
        exec.poststop += "jib destroy demojail";
 
        # Standard stuff
        exec.start += "/bin/sh /etc/rc";
        exec.stop = "/bin/sh /etc/rc.shutdown";
        exec.consolelog = "/var/log/jail_demojail_console.log";
        mount.devfs;          #mount devfs
        allow.raw_sockets;    #allow ping-pong
        devfs_ruleset="5";    #devfs ruleset for this jail
        allow.set_hostname = 1;
}

Step 6 – Turing on FreeBSD 12 jail service

Run the sysrc command:
# sysrc jail_enable=YES
Update/edit /etc/devfs.rules for the demojail:

Create /etc/devfs.rules

# vi /etc/devfs.rules
Append or edit the file as follows to allow /dev/tun access inside jail (see /etc/defaults/devfs.rules):
[devfsrules_jail_demojail=5] add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path 'tun*' unhide
add path 'bpf*' unhide
add path zfs unhide
Save and close the file.

Step 7 – Starting the jail service

Make sure we copy helper scripts using the cp command:
# cp -v /usr/share/examples/jails/{jib,jng} /usr/local/sbin/
Scripts installed:

/usr/share/examples/jails/jib -> /usr/local/sbin/jib
/usr/share/examples/jails/jng -> /usr/local/sbin/jng

We use the service command as follows to start/stop or get the status for jail service:
# service jail start
# service jail stop
# service jail restart
# service jail status

Step 8- Testing

First list all running jails, type:
# jls
Use jid or jailname as follows
# jexec 1 sh
# jexec demojail tcsh
Add a new user using the pw command:
# pw useradd -n vivek -G wheel -s /bin/tcsh -m -d /home/vivek
# passwd vivek
Update or install packages
# pkg update && pkg upgrade
# pkg install most
Verify networking:
# ifconfig
# sockstat -4
# ping -c 2 cyberciti.biz

Please note that e0b_demojail is my jail network interface.

Conclusion

And that is all for now. You learned how to configure a FreeBSD 12 jail with VNET along with ZFS. I suggest you read this page for more info and man pages by typing the following man command:
$ man jail
$ man devfs.conf