How to configure a FreeBSD Jail with vnet and ZFS

last updated in Categories ,

How do I install and configure a FreeBSD jail with vnet? How do I create FreeBSD jail with /etc/jail.conf without using iocage command or ezjail command line tool?

Introduction: Jails were introduced in FreeBSD 4.x by Poul-Henning Kamp. FreeBSD jail is nothing but operating system-level virtualization that allows partitioning a FreeBSD based Unix server. These mini systems called jails. Jails have their own root user and access rights. Jails can use network subsystem virtualization infrastructure or share existing network. FreeBSD jails are a powerful way to increase the security. Usually, you create jail per services such as web server, VPN server, database server and more. This page shows how to configure a FreeBSD Jail with vnet and ZFS. The following instructions tested on FreeBSD version 11.2.

How to configure a FreeBSD Jail with vnet and ZFS

The procedure to create a FreeBSD jail is as follows as of 11.2:

  1. Compile a FreeBSD kernel to include VIMAGE support
  2. Install jib and jng
  3. Create a zfs data set for basejail
  4. Configuring the jail.conf on the host
  5. Enable and start jail service

Let us see all steps in details to configure a FreeBSD Jail with vnet and ZFS.

Step 1. Configure/compile vnet kernel

You must compile FreeBSD kernel. You need a VIMAGE enabled FreeBSD kernel. You must have FreeBSD src installed in /usr/src/ directory. Use the cp command as follows to copy kernel config:
$ sudo cp -v /usr/src/share/examples/jails/VIMAGE /usr/src/sys/amd64/conf/VIMAGE
$ cd /usr/src/

Compile it:
$ sudo make KERNCONF=VIMAGE kernel
You can specify the maximum number of jobs that make may have running at any one time by passing the -j option. For example:
$ sudo make -j 16 KERNCONF=VIMAGE kernel
You must reboot the FreeBSD box:
$ sudo reboot
Verify new kernel version:
$ uname -v
FreeBSD 11.2-RELEASE #0: Wed Jul 11 21:35:37 IST 2018 root@nas04:/usr/obj/usr/src/sys/VIMAGE

Install required tools:
$ sudo cp -v /usr/src/share/examples/jails/{jib,jng} /usr/sbin/

Step 2. Create a zfs data set for jails

You need to set up the filesystem. Use the zpool command to get list of configured zfs:
# zpool list
Sample outputs:

NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
zroot  21.8T  68.7G  21.7T        -         -     0%     0%  1.00x  ONLINE  -

Type the following zfs command for zroot mounted at /jails/:
# zfs create -o mountpoint=/jails zroot/jails
Next create a zfs data set:
# zfs create zroot/jails/fullbasejail
Grab the FreeBSD 11.2 base files by visiting this page, and any other parts of FreeBSD you want in jails. Use the curl command or wget command or fetch command:
# cd /tmp
# wget https://download.freebsd.org/ftp/releases/amd64/11.2-RELEASE/base.txz
# wget https://download.freebsd.org/ftp/releases/amd64/11.2-RELEASE/lib32.txz

Use the tar command to unpack the base.txz and lib32.txz tarball fetched from FreeBSD’s ftp mirrors:
# tar -zxvf /tmp/11.2-RELEASE/base.txz -C /jails/fullbasejail
# tar -zxvf /tmp/11.2-RELEASE/lib32.txz -C /jails/fullbasejail

Update your FreeBSD base install

Apply updates to FreeBSD base install using the freebsd-update command:
# freebsd-update -b /jails/fullbasejail fetch install
Compare the system against a “known good” index of the installed release:
# freebsd-update -b /jails/fullbasejail IDS
Sample outputs:

Looking up update.FreeBSD.org mirrors... 3 mirrors found.
Fetching metadata signature for 11.2-RELEASE from update5.freebsd.org... done.
Fetching metadata index... done.
Fetching 1 metadata files... done.
Inspecting system... 
done.

Step 3. Configure the base jail

You just configured a base jail for quick deployment. One can clone basejail to create a new jail:
# zfs snapshot zroot/jails/fullbasejail@11.2-RELEASE-p0
List snapshots:
zfs list -t snapshot
Finally create a new jail named rsnapshot from the zfs snapshot:
# zfs send -R zroot/jails/fullbasejail@11.2-RELEASE-p0 | zfs receive zroot/jails/rsnapshot

How to configure basic jail settings

You need to make sure your jail has the right timezone, dns servers, a hostname, ip address and other stuff in rc.conf. Please note that you need to be in /jails/rsnapshot/ directory (adjust paths according to your setup):
# cd /jails/rsnapshot
Setup a dns by creating a file named etc/resolv.conf inside /jails/rsnapshot/:
# vi etc/resolv.conf
Append the following (set dns nameserver as per your setup):
nameserver 192.168.2.254
nameserver 192.168.2.18

Save and close the file in vim/vi text editor. Setup timezone for your FreeBSD jail using the ln command (I am setting up to IST):
# ln -v usr/share/zoneinfo/Asia/Kolkata etc/localtime
etc/localtime => usr/share/zoneinfo/Asia/Kolkata

Next you need to setup rc.conf inside /jails/rsnapshot/:
# vi etc/rc.conf
Sample settings:

# jail hostname #
host_hostname="rsnapshot"
 
# jail ip address and routing #
ifconfig_ng0_rsnapshot="inet 192.168.2.30 netmask 255.255.255.0"
defaultrouter="192.168.2.254"
 
# Start or stop services #
cron_flags="$cron_flags -J 15"
sendmail_enable="NONE"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
syslogd_flags="-c -ss"
ipv6_activate_all_interfaces="NO"
sshd_enable="YES"

Save and close the file.

How to configuring the jail.conf on the host

Edit or create the /etc/jail.conf as follows:
# vi /etc/jail.conf
Append the following config:

rsnapshot {
        host.hostname = "rsnapshot";   # hostname
        path = "/jails/rsnapshot";     # root directory
        exec.clean;
        exec.system_user = "root";
        exec.jail_user = "root";
        # ##########################################################################  
        # netgraph/vnet config info
        # ng0 is my vnet
        # idb1 is my physical network interface connected to the LAN (use ifconfig)
        # jng is located in /usr/sbin/
        # rsnapshot is my jail name
        # ##########################################################################  
        vnet; 
        vnet.interface = "ng0_rsnapshot";               # vnet interface(s)
        exec.prestart += "jng bridge rsnapshot igb1";   # bridge interface(s)
        exec.poststop += "jng shutdown rsnapshot";      # destroy interface(s)
 
        # Standard stuff
        exec.start += "/bin/sh /etc/rc";
        exec.stop = "/bin/sh /etc/rc.shutdown";
        exec.consolelog = "/var/log/jail_rsnapshot_console.log";
        mount.devfs;          #mount devfs
        allow.raw_sockets;    #allow ping-pong
        devfs_ruleset="5";    #devfs ruleset for this jail
        mount.devfs;
}

Save and close the file.

Turn on jail service

# sysrc jail_enable=YES

Create /etc/devfs.rules

# vi /etc/devfs.rules
Append or edit the file as follows to allow /dev/tun access inside jail (see /etc/defaults/devfs.rules):
[devfsrules_jail_rsnapshot=5]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path 'tun*' unhide
add path zfs unhide

Save and close the file.

Start the jail

# service jail start
# service jail status

How to login to my jail

To list running jails run:
# jls
Use jid or jailname as follows
# jexec 1
# jexec rsnapshot

Add a new user using the pw command:
# pw useradd -n vivek -G wheel -s /bin/tcsh -m -d /home/vivek
# passwd vivek

Update or install packages
# pkg update && pkg upgrade
# pkg install most

Verify networking:
# ifconfig
# sockstat -4
# ping -c 2 cyberciti.biz

How to configure a FreeBSD Jail with vnet and verify it
Please note that tun0 is my OpenVPN NIC running inside FreeBSD jail and ng0_rsnapshot is my jail NIC.

Conclusion

You just configure a FreeBSD Jail with vnet along with ZFS. I suggest you read this page for more info and man pages by typing the following man command:
$ man jail
$ man devfs.conf

Posted by: Vivek Gite

The author is the creator of nixCraft and a seasoned sysadmin, DevOps engineer, and a trainer for the Linux operating system/Unix shell scripting. Get the latest tutorials on SysAdmin, Linux/Unix and open source topics via RSS/XML feed or weekly email newsletter.

Notable Replies

  1. Yes, I am waiting for FreeBSD 12 host here. Mostly I will run on AWS and VM at AWS come default with UFS and directly connected to the Internet. I will write a new post covering that.

  2. Make sure you run ALL command as root user. It seems command ngctl not found. Can you see ngctl?

    type ngctl
    

    It shows path:

    ngctl is hashed (/usr/sbin/ngctl)
  3. This tutorial created for FreeBSD 11.x. So it won’t work on FreeBSD 12. There are some changes.

  4. I am planning to write a new one as there are many changes between 11 and 12 version. Stay tuned.

  5. Hello Vivek Gite;
    Been trying to get vnet jails to work on a gateway host. Having problems with vnet network access to the public internet. I need some help. From this article I can tell you have the networking knowledge that can help me figure out what is wrong with my vnet configuration. I wrote my problem as a draft for an article that you may use to post on this site. I can be reached at joeb_722 at yahoo dot com

    This is an example of a vnet jail configuration that is run on a real hardware computer having a network interface connected to a public internet service provider. FreeBSD 12.0 is the operating system running on this hardware.

    VIMAGE which is required for vnet jails is included in the FreeBSD base kernel in this release, and the PF firewall and its NAT (network address translation) function have also been adapted to be vnet jail aware. A vnet jail requires NAT to access the public internet.

    The bridge/epair method is used to connect the vnet jails network to the hosts network. The setting up and the tearing down of the vnet jails bridge/epair is done in the vnet jails jail.conf file.

    The hosts "service jail" command is used to start/stop the vnet jails. The setup configurations are further separated into all the vnet jails being defined in a single jail.conf file or each jail definition being in a separate jail.conf file.


    Information on the sysrc command and the "service jail" command.

    The sysrc command adds or changes statements in the hosts /etc/rc.conf file.
    Additions are placed at the bottom of the /etc/rc.conf file.

    $ sysrc jail_enable=YES
    Used to enable the "service jail" command. Will auto start jails at boot time and auto stop jails at halt & shutdown time.

    $ sysrc jail_list=""
    Used by the "service jail" command. Empty list means all jails are in a single /etc/jail.conf file.

    $ sysrc jail_list="jail-name jail-name …"
    Used by the "service jail" command. Means search /etc for jail.xxx.conf files where xxx is jail-name from the list.

    This "service jail" command setup works for both vnet jails and non-vnets.

    Format of service command

    $ service jail start
    $ service jail restart
    $ service jail stop
    

    or selectively;

    $ service jail start jail-name
    $ service jail restart jail-name
    $ service jail stop jail-name
    

    Creating vnet jails directory tree the easy way.

    mkdir -p /usr/local/jails
    cd /usr/local/jails
    fetch -avrA "http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.0-RELEASE/base.txz"

    Repeat the following 2 commands changing the jail-name to create as many vnet jails as you want.

    mkdir jail-name
    xzdec base.txz | tar --unlink -xpJf - -C "jail-name"

    Copy host files needed by the jail.

    cp /etc/resolv.conf jail-name/etc
    cp /etc/localtime jail-name/etc
    

    Preparing the host’s /etc/rc.conf for jails.

    Sample 1: Single jail.conf(5) method.
    All jail definitions in single jail.conf file.
    On the host system console issue following commands one time.

    sysrc jail_enable=YES
    sysrc jail_list=""

    Copy the jail.conf statements shown below and paste into /etc/jail.conf
    Edit the /etc/jail.conf following the edit instructions below.

    Sample 2: Per-jail jail.conf(5)
    Each jail definition in individual jail.jail-name.conf file.
    On the host system console issue following commands one time.

    sysrc jail_enable=YES
    sysrc jail_list+="jail-name jail-name jail-name ..."

    Copy the jail.conf statements shown below and paste
    into /etc/jail.jail-name.conf
    Edit the /etc/jail.jail-name.conf following the edit instructions below.


    This is the example vnet jail jail.conf content.

    jail-name {
    host.hostname = "jail-name";
    path = "/usr/local/jails/jail-name";
    exec.consolelog = "/var/log/jail.jail-name.console.log";
    mount.devfs;
    devfs_ruleset = "70";
    vnet = "new";
    vnet.interface = "epair15b";
    exec.prestart = "ifconfig bridge15 create up";
    exec.prestart += "ifconfig epair15 create up";
    exec.prestart += "ifconfig bridge15 addm epair15a";
    exec.prestart += "ifconfig bridge15 addm re0";
    exec.start = "ifconfig epair15b inet 10.0.110.25/24";
    exec.start += "route add default 10.0.110.2";
    exec.start += "/bin/sh /etc/rc";
    exec.stop = "/bin/sh /etc/rc.shutdown";
    exec.poststop = "ifconfig epair15a destroy";
    exec.poststop += "sleep 2";
    exec.poststop += "ifconfig bridge15 destroy";
    }
    

    Standard edit instructions for the jail.conf file.

    Substitute all 4 places where the word jail-name is shown with the real jail name created above.
    Vnet jails must have an ip address assigned to it. Change the ip address 10.0.110.25/24 to what ever ip address you’re using on your host system leaving the /24 suffix.

    All vnet jails must have a route pointing to the hosts default gateway ip address assigned by the ISP. Issue "route -n get default" command. Take the gateway ip address shown in the output display and substitute it for the 1.2.3.4 ip address. ???

    The re0 in the exec.prestart line represents the interface name of the interface facing the public internet. Replace re0 with the interface name facing your public internet. You can determine the interface name to use from the "route -n get default" command issued on the host.


    Setup PF firewall to run inside of vnet jail.

    Copy the following and paste it into the /etc/rc.conf file of the vnet jail you’re setting up.

    gateway_enable="YES"
    pf_enable="YES"
    pf_rules="/etc/pf.conf"
    pflog_enable="YES"
    pflog_logfile="/var/log/pflog"
    ftpproxy_enable="NO"
    

    Pre-allocate the pflog file
    touch jail-name/var/log/pflog


    Vnet jails need devfsrules to enable pf firewall to run.

    On the host create /etc/devfs.rules file and populate it with the following content. Then issue the "service devfs restart" command to enable this new rule.

    This only needs to be done one time.

    devfsrules for pf to function in a vnet jail.

    [vnet_pf=70]
    add include $devfsrules_hide_all
    add include $devfsrules_unhide_basic
    add include $devfsrules_unhide_login
    add include $devfsrules_jail
    add path ‘bpf*’ unhide
    add path pf unhide
    add path pflog unhide


    Copy the following and paste it into the /etc/pf.conf file of the vnet jail you’re setting up. This is very simple PF rules, pass every thing in and out and log, so traffic can be viewed.

    set block-policy drop
    set fail-policy drop
    set state-policy if-bound
    scrub in on epair15b all
    set skip on lo0
    nat on epair15b from 10.0.110.25 to any -> host-public-ip
    pass out log (all) quick on epair15b from any to any
    pass in log (all) quick on epair15b from any to any


    PF information

    PF is now a loadable module. If PF is used as the hosts firewall than nothing further needs to be done. If the host is running IPFW or IPFILTER firewall then PF has to be loaded into the kernel one time per boot using this command "kldload pf.ko pflog.ko" or added to boot/loader.conf. To see what modules have been loaded issue kldstat -v. To unload the PF modules
    use "kldunload -f pf.ko pflog.ko" command.


    Testing your vnet jail.

    From the hosts console.
    service jail start
    service jail stop

    It may take up to 30 seconds for the vnet jail to complete starting.
    You can issue the "jls" command to verify the jail is running.
    To log into the running jail issue "jexec jail-name login -f root"


Continue the discussion www.nixcraft.com

12 more replies

Participants

Historical Comment Archive

18 comment

    Still, have a question? Get help on our forum!