Self Hosted

Self Hosting: Episode 1 - Hypervisor

Michael Graves

Sep 7, 2025 • 5 min read

Software Used:

Introduction

Before we can deploy any services we need two things. First a place to store those services and two a place to run those services from (i.e storage and compute). These two requirements do not need to run on the same system and for larger self hosted sites or businesses, they probably wont. But for these examples I will present them using a single system.

In these examples I will be utilizing FreeBSD as my compute and storage system. There are other systems out there: Proxmox, XCP-NG, etc. Please feel free to use any/all of these. My preference is FreeBSD because it gives me all the underlying services I need in a rock solid operating system.

Installing Base OS

Installing FreeBSD is straight forward. Chapter 2 of the FreeBSD handbook will walk you through installing FreeBSD on a bare system.

Network Prep

Within the hypervisor we will create separate virtual networks. Each virtual network will be on a separate VLAN. The hypervisor is not required to have an interface on any of the virtual networks. In fact, we don't want it to.

Create Bridge for Virtual Networks

On FreeBSD (as in Linux) a bridge network interface is used to create the virtual network. These commands will create the bridge and rename it to something more meaningful.

ifconfig bridge1 create
ifconfig bridge1 name net-common
ifconfig bridge2 create
ifconfig bridge2 name net-iot
ifconfig bridge3 create
ifconfig bridge3 name net-dmz
ifconfig bridge4 create
ifconfig bridge4 name net-external

Attach each bridge to the appropriate VLAN. Creating VLANs allows this system to communicate with other systems while maintaining the segments. For this example I will assume that the physical interface of the hypervisor is "vmx0". Replace this with the actual physical interface name.

ifconfig vlan10 create
ifconfig vlan10 vlan 10 vlandev vmx0
ifconfig net-common addm vlan10
ifconfig vlan20 create
ifconfig vlan20 vlan 20 vlandev vmx0
ifconfig net-iot addm vlan20
ifconfig vlan30 create
ifconfig vlan30 vlan 30 vlandev vmx0
ifconfig net-dmz addm vlan30
ifconfig vlan90 create
ifconfig vlan90 vlan 90 vlandev vmx0
ifconfig net-external addm vlan90

Next we will update /etc/rc.conf to make sure the changes will take affect on next reboot.

sysrc ifconfig_vlan10="vlan 10 vlandev vmx0
sysrc ifconfig_bridge1="name net-common addm vlan10"
sysrc ifconfig_vlan20="vlan 20 vlandev vmx0
sysrc ifconfig_bridge2="name net-iot addm vlan20"
sysrc ifconfig_vlan30="vlan 30 vlandev vmx0
sysrc ifconfig_bridge3="name net-dmz addm vlan30"
sysrc ifconfig_vlan90="vlan 90 vlandev vmx0
sysrc ifconfig_bridge4="name net-external addm vlan90"

sysrc cloned_interfaces+=" bridge1 vlan10"
sysrc cloned_interfaces+=" bridge2 vlan20"
sysrc cloned_interfaces+=" bridge3 vlan30"
sysrc cloned_interfaces+=" bridge4 vlan90"

Storage Prep

FreeBSD comes with ZFS. If you are not familiar with it see here, here and here for explanations. ZFS provides the many features we will need to hosting our environment.

Create data_dirs

This environment will have data that we want to protect. While ZFS will do everything reasonable to keep the data safe that isn't a 100% guarantee. Setting up our data_dirs will help us determine what data we consider important.

zpool create data01 da2

Now we create a dataset for virtual machine images

zfs create data01/vms

Now we create a dataset for jail'd processes.

zfs create data01/jails

Create a dataset for shared user data.

zfs create data01/user_data

Finally we create a dataset for backups

zfs create data01/backup_data

Virtual machines

Virtual machines can take several different flavors. They can be fully emulated hardware (type 1/2) or just isolated processes in containers (type 3). In this environment both will be used. To start with we will make use of FreeBSD Jails.

Prepare for Jails

To prepare the server for jails add a /etc/jail.conf file

#
.include "/etc/jail.conf.d/*.conf";

/etc/jail.conf

And then enable the jails to start on boot.

service jails enable

One nice feature of ZFS is that it can do snapshots. A snapshot is a exact copy of a dataset at the moment the snapshot is taken. Any updates to the dataset will not update the snapshot. We will use this feature of ZFS to create a snapshot of a jail. Then we will use that jail snapshot when we create new jails. By doing this we make adding new jail very quick.

To prepare the jail template first we create a new dataset.

zfs create -p data01/jails/templates/14.2-RELEASE

Next we will download the FreeBSD base archive and extract into the new dataset.

fetch -o - https://download.freebsd.org/ftp/releases/amd64/amd64/14.2-RELEASE/base.txz | tar -xf - -C /data01/jailes/templates/14.2-RELEASE --unlink

Update the resolv.conf and localtime settings.

cp /etc/resolv.conf /data01/jails/templates/14.2-RELEASE/etc/resolv.conf
cp /etc/localtime /data01/jails/templates/14.2-RELEASE/etc/localtime

Optimally we can perform any OS updates between now and when the base.txz was originally created.

freebsd-update -b /data01/jails/templates/14.2-RELEASE fetch install

Now take a snapshot of the dataset

zfs snapshot data01/jails/templats/14.2-RELEASE@base

An when we are ready to deploy a new jail we will "clone" the dataset for our jail.

zfs clone data01/jails/templates/14.2-RELEASE@base data01/jails/containers/<jailname>

Prepare for Virtual Machines

The virtual machines using Bhyve (the FreeBSD hypervisor) takes a few more steps. First install some packages

pkg install -y vm-bhyve bhyve-firmware edk2-bhyve grub-bhyve

After the packages are installed update the rc.conf with the following variables

sysrc vm_delay=5
sysrc vm_dir=zfs:/data01/vms

Enable the VMs to start on boot

service vm enable

Initialize the VM configuration

/usr/local/sbin/vm init

And finally update the VM config file adding the different segments so they were be available to any VMs.

vm switch create -t manual -b net-iot net-iot
vm switch create -t manual -b net-dmz net-dmz
vm switch create -t manual -b net-common net-common
vm switch create -t manual -b net-external net-external

Now the hypervisor is ready for VMs

Backups

While ZFS is incredibly stable and will do everything it can to protect your data, problems can happen outside of ZFS's control. For those case we need to create a backup. (See discussion on 3-2-1 backup scheme: here, here, here). Perhaps the best tool I've found to automate these backups is Sanoid/Syncoid.

Sanoid

Sanoid is a script that can be called from cron that will create and manage snapshots automatically. This will allow us to protect the files on the filesystem from accidental deletion. However we are not fully protected. If something were to happen to the hose or to enough hard drives all data could be lost. So we need to copy the data off of this system to someplace else.

First install sanoid on the system

pkg install sanoid

Next configure the sanoid.conf file located in /usr/local/etc/sanoid directory

#
[ data01/vms ]
        use_template = production
        recursive = yes

[ data01/jails ]
        use_template = production
        recursive = yes

[ data01/data ]
        use_template = production

#############################
# templates below this line #
#############################

# name your templates template_templatename. you can create your own, and use them in your module definitions above.

[template_demo]
        daily = 60

[template_production]
        frequently = 0
        hourly = 36
        daily = 30
        monthly = 3
        yearly = 0
        autosnap = yes
        autoprune = yes

/usr/local/etc/sanoid/sanoid.conf

Finally update crontab using 'crontab -e' command add the sanoid command.

*/15 * * * * TZ=UTC /usr/local/bin/sanoid --cron

Syncoid

Syncoid is also a script run from cron. It will take the snapshots created by Sanoid and ZFS send the snapshots to another system running ZFS. This systems does not need to be FreeBSD.

Syncoid uses the sanoid.conf file to determine what to sync so we only need to specify the destination server. Update cron and specify bkupuser, bkuphost, and bkuppath to tell syncoid where to sync the snapshots.

0 1 * * * TZ=UTC /usr/local/bin/syncoid --no-privilege-elivation --create-bookmark data01/vms <bkupuser>@<bkuphost>:<backuppath>
15 1 * * * TZ=UTC /usr/local/bin/syncoid --no-privilege-elivation --create-bookmark data01/jails <bkupuser>@<bkuphost>:<backuppath>
30 1 * * * TZ=UTC /usr/local/bin/syncoid --no-privilege-elivation --create-bookmark data01/data <bkupuser>@<bkuphost>:<backuppath>

Conclusion

With the physical systems out of the way, we can now begin creating the different services that will be needed to setup our self hosted environment.