1
2
3
4

[x] BHYVE_FIRMWARE  Required to run UEFI guests
[x] EXAMPLES        Install example guest templates
[x] GRUB2_BHYVE     Required to run any guests that need a Grub bootloader
[x] TMUX            Tmux console access instead of cu/nmdm

Configuration

vm-bhyve

Fist we need to create a directory where vm-bhyve will store it’s configuration as well as the various VMs created:

1
2

zfs create -o mountpoint=/var/bhyve main/bhyve  # Create vm-bhyve directory
zfs set compress=zstd main/bhyve                # Enable compression if desired

Enabling vm-bhyve and specifying it’s directory is done in the system configuration file rc.conf:

1
2

vm_enable="YES"
vm_dir="zfs:main/bhyve"

Allows vm-bhyve to initialize it’s infrastructure:

1

vm init

Various templates are available in /usr/local/share/examples/vm-bhyve/, they should be copied (and adapted) in the .templates directory inside the defined in vm_dir (in case of a zfs filesystem, the mountpoint is used).

1

cp /usr/local/share/examples/vm-bhyve/* /var/bhyve/.templates/

Networking

The bridge interface on which bhyve will attach is manually configured, instead of letting vm-bhyve manage it, so it’s easier to integrate on other part of the system.

Create the vmnet0 interface and assign it an IP address, the interface is also given a group name jail to ease management in complex firewall rules:

1
2
3

cloned_interfaces="bridge1"
ifconfig_bridge1_name="vmnet0"
ifconfig_vmnet0="inet 192.168.100.1/24 group jail"

If we want to give the VM access to the outside world we need to have the host act as a gateway, and also as here the VMs have been defined on a private network NAT must be enabled:

1

nat pass on wan0 inet from vmnet0:network to any -> wan0

1
2

gateway_enable="yes"
pf_enable="yes"

It is possible to use, for a more complex configuration, a full featured DHCP, TFTP and DNS server to provide the necessary network services for booting and network access, but we have chosen here to use dnsmasq which conveniently provide all these services with a simple configuration:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

# Select listening interface
interface=vmnet0        # Select the interface on which to listen
except-interface=lo0    # Ensure we don't listen on loopback 
bind-interfaces         # Bind to interfaces instead of listening on wildcard

# DNS (with DNSSEC enabled)
domain-needed
conf-file=/usr/local/share/dnsmasq/trust-anchors.conf
dnssec
dnssec-check-unsigned

# DHCP
dhcp-authoritative
dhcp-range=192.168.100.10,192.168.100.150,24h
# Using statically allocated addresses for some guests
#  infinite = maximum lease time
dhcp-host=debian,192.168.100.55                      # from provided hostname
dhcp-host=58:9c:fc:0d:84:bd,192.168.100.52,infinite  # from MAC address

We now let vm-bhyve know that we have created an interface where it will be able to connect the VMs:

1
2
3
4
5
6

# List of available switch
switch_list="bhyve-bridge"

# Switch definition
type_bhyve-bridge="manual"
bridge_bhyve-bridge="vmnet0"

1
2
3
4

options {
        listen-on       { 192.168.1.1; 192.168.100.1 };
        listen-on-v6    { none; };
}

VM

Templates

As we are using zfs, the various templates will be adapted to use a sparse zvol as disk, and it will be named root

1
2

disk0_name="root"
disk0_dev="sparse-zvol"

Console

There is a virtio console, which provide a socket where to connect (with socat for example), but:

• there’s no way to use console port feature (➞ no /dev/hvc? in linux)
• emergency write is advertised, but is a no-op

For a true console, we need to use vm console which connect (defined when configuring the VM) to an emulated serial line:

• from null-modem using cu and nmdm device
• from stdio/stdout using tmnux

Shared directory

• Only available in FreeBSD 13.
• This option is not directly handled by vm command, the configuration file vm.conf need to be edited.
• A free PCI slot (option -s) need to be selected, check vm-bhyve.log to see what was assigned.

1

bhyve_options="-s 15,virtio-9p,foo=/path/to/foo,bar=/path/to/bar"

Sound

• Only available in FreeBSD 13.
• This option is not directly handled by vm command, the configuration file vm.conf need to be edited.
• A free PCI slot (option -s) need to be selected, check vm-bhyve.log to see what was assigned.

1

bhyve_options="-s 16,hda,play=/dev/dsp,rec=/dev/dsp"

Entropy driver

To supply high-quality randomness from the hypervisor to the guest:

1

virt_random="yes"

The guest will also need a specific driver:

• FreeBSD: virtio_random
• Windows: see virtio-win

Guests

Windows

Executing command at startup:

1
2
3

[HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Run]
"F Drive"="subst F: D:\\mount\\db"
"G Drive"="subst G: D:\\mount\\log"

Linux

Console

The virtio console is available through named devices in /dev/virtio-ports/ directory (which hold seem links to numbered device /dev/vport?p?).

1
2

usermod -a -G dialout johndoe
usermod -a -G tty johndoe

Shared directory

1
2

# See: https://www.kernel.org/doc/html/latest/filesystems/9p.html
mount -t 9p -o trans=virtio,cache=mmap,msize=512000 sharename /path/to/mountpoint

Commands

1
2

vm create -t debian debian  # Create VM from template
vm configure debian         # Perform extra configuration

1
2

debian_cd_repos_url=https://cdimage.debian.org/debian-cd/current/amd64/iso-cd
vm iso ${debian_cd_repos_url}/debian-10.9.0-amd64-netinst.iso

1

vm install debian debian-10.9.0-amd64-netinst.iso

1

vm start debian

1

vm console debian

1
2

vm stop debian        # Ask for gracefull shutdown
vm poweroff debian    # Terminate immeditately

Troubleshouting

All ports busy (vm console)

# vm console debian
all ports busy

There is a cu process already connected to the console, find it and kill it