Today I will be creating a OpenBSD guest via bhyve on OmniOS. I will also be adding a Pass Through Ethernet Controller so I can have a multi-homed guest that will serve as a firewall/router.
This post will cover setting up bhyve on OmniOS, so it will also be a good introduction to bhyve. As well, I look into OpenBSD’s uEFI boot loader so if you have had trouble with this, then you are in the right place.
I will cover installing bhyve onto an OmniOS system briefly here. If you already have bhyve installed, you can jump ahead to Setting up the OpenBSD boot disk.
To install bhyve onto OmniOS, enter the command below:
pkg install brand/bhyve
This will also install
Create storage for the OmniOS zone guests
Every zone on a system has at least one dedicated ZFS dataset and it’s good practice to create a parent dataset under where all zones will live. A common convention is to mount this dataset as /zones and create it at the top level of a pool.
I will create a zpool named
data, on disk device
# zpool create data c2t1d0
This can be verified with the following command:
# zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data 928G 526K 928G - - 0% 0% 1.00x ONLINE - rpool 232G 7.98G 224G - - 0% 3% 1.00x ONLINE -
Now that this zpool exists, the following command will create the top-level ZFS dataset, named
# zfs create -o mountpoint=/zones data/zones
The creation of the ZFS dataset can be verified with the following command:
# zfs list NAME USED AVAIL REFER MOUNTPOINT data 153K 899G 24K /data data/zones 24K 899G 24K /zones rpool 15.4G 209G 26K /rpool rpool/ROOT 1001M 209G 24K legacy rpool/ROOT/omnios-r151034 44.5M 209G 620M legacy ...
That is all there is to setting up your bhyve environment!
Setting up the OpenBSD boot disk
miniroot67.fs disk image:
# mkdir /zones/iso # cd /zones/iso # wget https://cdn.openbsd.org/pub/OpenBSD/6.7/amd64/miniroot67.fs
Create a block device to boot from:
# lofiadm -a /zones/iso/miniroot67.fs /dev/lofi/1
Create a virtual network interface card (vnic)
Create a vnic with the following command:
# dladm create-vnic -l e1000g0 openbsd0
Note: I use the
hostname as the zone name for the vnic.
The new vnic can be verifed with the following command:
# dladm show-link LINK CLASS MTU STATE BRIDGE OVER e1000g0 phys 1500 up -- -- openbsd0 vnic 1500 up -- e1000g0
Configure the physical network card for PCI-Passthru
prtconf to find the device we will be adding to OpenBSD:
# prtconf -dD ... ... pci8086,8c18 (pciex8086,8c18) [Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5], instance #0 (driver name: pcieb) pci8086,1 (pciex8086,1521) [Intel Corporation I350 Gigabit Network Connection], instance #1 (driver name: igb) pci8086,1 (pciex8086,1521) [Intel Corporation I350 Gigabit Network Connection], instance #2 (driver name: igb) pci8086,1 (pciex8086,1521) [Intel Corporation I350 Gigabit Network Connection], instance #3 (driver name: igb) pci8086,1 (pciex8086,1521) [Intel Corporation I350 Gigabit Network Connection], instance #4 (driver name: igb) ... ...
I will be adding the I350 Gigabit Network device, found and listed above in the output from
We need to convert this into a
ppt device and this can be done by adding the PCI Identifiers to the
ppt_matches files, as follows:
# echo 'ppt "pciex8086,1521"' >> /etc/ppt_aliases # echo 'pciex8086,1521' >> /etc/ppt_matches
For the system to attach these as ppt devices a reboot is necessary. Once your system has come back online, you can verify that the ppt devices are attached:
# pptadm list -a DEV VENDOR DEVICE PATH /dev/ppt1 8086 1521 /pci@0,0/pci8086,8c18@1c,4/pci8086,1@0 /dev/ppt2 8086 1521 /pci@0,0/pci8086,8c18@1c,4/pci8086,1@0,1 /dev/ppt3 8086 1521 /pci@0,0/pci8086,8c18@1c,4/pci8086,1@0,2 /dev/ppt4 8086 1521 /pci@0,0/pci8086,8c18@1c,4/pci8086,1@0,3
Now these devices are ready to be passed through to bhyve
Create the ZFS dataset to store the bootdisk for the guest
This is the ZFS dataset that will be the storage device for the OpenBSD bhyve zone. Create the ZFS dataset as follows:
omnios# zfs create -V 10G data/openbsd
Configure the OpenBSD bhyve zone
Zones are configured on OmniOS via the
zonecfg command, this is also true for bhyve zones.
The following can be pasted into
zonecfg once it has been executed. You will need to change any directives that are specific to your system.
# zonecfg -z openbsd openbsd: No such zone configured Use 'create' to begin configuring a new zone. create -b set brand=bhyve add attr set name=type set type=string set value=openbsd end set zonepath=/zones/openbsd set ip-type=exclusive add net set allowed-address=192.168.1.67/24 set physical=openbsd0 end add device set match=/dev/lofi/1 end add attr set name=disk set type=string set value=/dev/lofi/1 end add device set match=/dev/zvol/rdsk/data/openbsd end add attr set name=bootdisk set type=string set value=/dev/zvol/rdsk/data/openbsd end add attr set name=acpi set type=string set value=off end add attr set name=bootrom set type=string set value=BHYVE_RELEASE end add device set match=/dev/ppt1 end add attr set name=extra set type=string set value="-S -s 8:0,passthru,/dev/ppt1" end verify commit exit
The above is quite intuitive and should not need any explanation.
Install the OpenBSD bhyve zone
Once a new zone has been configured with
zonecfg it needs to be installed. This is achieved with the following command:
# zoneadm -z openbsd install
Boot the OpenBSD bhyve zone
Everything is now set and we can boot the OpenBSD bhyve zone as follows:
# zoneadm -z openbsd boot
You now have 5 seconds to access the OpenBSD boot loader. After this time, you will not be able to access the OpenBSD bhyve zone via the console. Or at all… that bird will have flown and you will have to reboot the zone.
Enter the console on the OpenBSD bhyve zone
Configuration of the OpenBSD guest needs to be done via the console, as follows:
# zlogin -C openbsd [Connected to zone 'openbsd' console]
Immeadiatley after entering the console, press the space bar & then the enter key to stop OpenBSD booting into the VGA Output mode.
Setting the console in the OpenBSD bhyve zone
After successfully accessing the console, the
boot> prompt will appear. The OpenBSD guest now needs to be configured to redirect all output to the
com0 serial port.
boot> set tty com0
Boot the OpenBSD bhyve zone
com0 serial console is set, the OpenBSD guest can be booted:
OpenBSD can now be installed as normal. The Pass-Thru Ethernet Device will show in the network configuration as a normal device and no special care needs to be taken.
Where it gets tricky is the Disk Creation section, which I cover next
Setting up the root disk with uEFI
For me, GPT is the sure-fire way to make sure uEFI is installed… all the time… every time. You will see a warning about uEFI & GPT, but from my experiences, this can safley be ignored. And besides, we will do a full check after installation just to make sure uEFI is where it is supposed to be.
So when you reach that part of the installation that deals with the disk, we want to be installing onto
sd0 and the creation looks a little something like this:
Available disks are: sd0 sd1. Which disk is the root disk? ('?' for details) [sd0] sd0 No valid MBR or GPT. Use (W)hole disk MBR, whole disk (G)PT or (E)dit? [whole] G An EFI/GPT disk may not boot. Proceed? [no] yes Setting OpenBSD GPT partition to whole sd0...done. The auto-allocated layout for sd0 is: # size offset fstype [fsize bsize cpg] a: 1218736.0K 1024 4.2BSD 2048 16384 1 # / b: 262144.0K 2438496 swap c: 10485760.0K 0 unused d: 3145728.0K 2962784 4.2BSD 2048 16384 1 # /usr e: 2097152.0K 9254240 4.2BSD 2048 16384 1 # /home i: 480.0K 64 MSDOS Use (A)uto layout, (E)dit auto layout, or create (C)ustom layout? [a] /dev/rsd0a: 1190.2MB in 2437472 sectors of 512 bytes 6 cylinder groups of 202.50MB, 12960 blocks, 25920 inodes each /dev/rsd0e: 2048.0MB in 4194304 sectors of 512 bytes 11 cylinder groups of 202.50MB, 12960 blocks, 25920 inodes each /dev/rsd0d: 3072.0MB in 6291456 sectors of 512 bytes 16 cylinder groups of 202.50MB, 12960 blocks, 25920 inodes each Available disks are: sd0. Which disk do you wish to initialize? (or 'done') [done] /dev/sd0a (a3b707add095aa86.a) on /mnt type ffs (rw, asynchronous, local) /dev/sd0e (a3b707add095aa86.e) on /mnt/home type ffs (rw, asynchronous, local, nodev, nosuid) /dev/sd0d (a3b707add095aa86.d) on /mnt/usr type ffs (rw, asynchronous, local, nodev)
Continue on with installing the file sets via http and when that is all finished, we will make those checks for the uEFI boot loader.
Checking the root disk for uEFI
Exit out of the installer and into the shell and from here we can verify all the right uEFI bits are in place:
CONGRATULATIONS! Your OpenBSD install has been successfully completed! When you login to your new system the first time, please read your mail using the 'mail' command. Exit to (S)hell, (H)alt or (R)eboot? [reboot] S To boot the new system, enter 'reboot' at the command prompt. openbsd# fdisk sd0 Disk: sd0 Usable LBA: 64 to 20971456 [20971520 Sectors] #: type [ start: size ] ------------------------------------------------------------------------ 1: EFI Sys [ 64: 960 ] 3: OpenBSD [ 1024: 20970433 ]
To be absolutely certain, we can check inside the uEFI partiton to make sure the
bootx64.efi file is available.
openbsd# mount /dev/sd0i /mnt2 openbsd# ls /mnt2/efi/boot/ bootia32.efi bootx64.efi openbsd# umount /mnt2
Reboot the system
We can issue the reboot command and our new OpenBSD bhyve zone will come up cleanly:
# reboot >> OpenBSD/amd64 BOOTX64 3.50 boot> booting hd0a:/bsd: 12879176+2753552+324816+0+704512 [813343+128+1022112+747816]=0x125cf68 entry point at 0x1001000 [ using 2584424 bytes of bsd ELF symbol table ] Copyright (c) 1982, 1986, 1989, 1991, 1993 The Regents of the University of California. All rights reserved. Copyright (c) 1995-2020 OpenBSD. All rights reserved. https://www.OpenBSD.org OpenBSD 6.7 (GENERIC) #179: Thu May 7 11:02:37 MDT 2020 email@example.com:/usr/src/sys/arch/amd64/compile/GENERIC real mem = 1056288768 (1007MB) ... ... ... ...
Escaping from the console
To escape from the console, use
~~. You will be brought back to the OmniOS system.
Post installation tasks
There is nothing necessary to be done, however a little tidying up is always good.
This should be done when you have the OpenBSD bhyve zone in a shut-off state.
# zonecfg -z openbsd remove device match=/dev/lofi/1 remove attr name=disk verify commit exit
lofi block device:
lofiadm -d /dev/lofi/1
Connecting to the installed OpenBSD bhyve zone
Console and optionally
ssh should be available to connect to the zones:
# zlogin -C openbsd
ssh into the OpenBSD bhyve zone if you have configured
There are quite a few ways to configure zones, this has been one of them. Some other tips that you may find easier are as follows:
zonecfg, you can use zone templates to reduce typing a bit, so
create -t bhyvein this case. Then you don’t need to set
- In the
netsection, you can set
global-nicwhich makes the vnic get created and deleted automatically with VM boot.
- There is also the options to use
zcagein assisting you with the setup of Zones on OmnioOS.
Thanks to Andy at citrus-it for these tips.