OpenBSD guest with bhyve - OmniOS

Share on:

Today I will be creating a OpenBSD guest via bhyve on OmniOS. I will also be adding a Pass Through Ethernet Controller so I can have a multi-homed guest that will serve as a firewall/router.

This post will cover setting up bhyve on OmniOS, so it will also be a good introduction to bhyve. As well, I look into OpenBSD's uEFI boot loader so if you have had trouble with this, then you are in the right place.

Install bhyve

I will cover installing bhyve onto an OmniOS system briefly here. If you already have bhyve installed, you can jump ahead to Setting up the OpenBSD boot disk.

To install bhyve onto OmniOS, enter the command below:

1pkg install brand/bhyve

This will also install system/bhyve. Simple!

Create storage for the OmniOS zone guests

Every zone on a system has at least one dedicated ZFS dataset and it’s good practice to create a parent dataset under where all zones will live. A common convention is to mount this dataset as /zones and create it at the top level of a pool.

I will create a zpool named data, on disk device c2t1d0

1# zpool create data c2t1d0

This can be verified with the following command:

1# zpool list
2NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
3data    928G   526K   928G        -         -     0%     0%  1.00x  ONLINE  -
4rpool   232G  7.98G   224G        -         -     0%     3%  1.00x  ONLINE  -

Now that this zpool exists, the following command will create the top-level ZFS dataset, named zones:

1# zfs create -o mountpoint=/zones data/zones

The creation of the ZFS dataset can be verified with the following command:

1# zfs list
2NAME                                 USED  AVAIL  REFER  MOUNTPOINT
3data                                 153K   899G    24K  /data
4data/zones                            24K   899G    24K  /zones
5rpool                               15.4G   209G    26K  /rpool
6rpool/ROOT                          1001M   209G    24K  legacy
7rpool/ROOT/omnios-r151034           44.5M   209G   620M  legacy
8...

That is all there is to setting up your bhyve environment!

Setting up the OpenBSD boot disk

Download OpenBSD miniroot67.fs disk image:

1# mkdir /zones/iso
2# cd /zones/iso
3# wget https://cdn.openbsd.org/pub/OpenBSD/6.7/amd64/miniroot67.fs

Create a block device to boot from:

1# lofiadm -a /zones/iso/miniroot67.fs 
2/dev/lofi/1

Create a virtual network interface card (vnic)

Create a vnic with the following command:

1# dladm create-vnic -l e1000g0 openbsd0

Note: I use the hostname as the zone name for the vnic.

The new vnic can be verifed with the following command:

1# dladm show-link
2LINK        CLASS     MTU    STATE    BRIDGE     OVER
3e1000g0     phys      1500   up       --         --
4openbsd0    vnic      1500   up       --         e1000g0

Configure the physical network card for PCI-Passthru

First search prtconf to find the device we will be adding to OpenBSD:

 1# prtconf -dD
 2...
 3...
 4pci8086,8c18 (pciex8086,8c18) [Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5], instance #0 (driver name: pcieb)
 5            pci8086,1 (pciex8086,1521) [Intel Corporation I350 Gigabit Network Connection], instance #1 (driver name: igb)
 6            pci8086,1 (pciex8086,1521) [Intel Corporation I350 Gigabit Network Connection], instance #2 (driver name: igb)
 7            pci8086,1 (pciex8086,1521) [Intel Corporation I350 Gigabit Network Connection], instance #3 (driver name: igb)
 8            pci8086,1 (pciex8086,1521) [Intel Corporation I350 Gigabit Network Connection], instance #4 (driver name: igb)
 9...
10...

I will be adding the I350 Gigabit Network device, found and listed above in the output from prtconf -dD.

We need to convert this into a ppt device and this can be done by adding the PCI Identifiers to the ppt_aliases & ppt_matches files, as follows:

1# echo 'ppt "pciex8086,1521"' >> /etc/ppt_aliases
2# echo 'pciex8086,1521' >> /etc/ppt_matches

For the system to attach these as ppt devices a reboot is necessary. Once your system has come back online, you can verify that the ppt devices are attached:

1# pptadm list -a
2DEV        VENDOR DEVICE PATH
3/dev/ppt1  8086   1521   /pci@0,0/pci8086,8c18@1c,4/pci8086,1@0
4/dev/ppt2  8086   1521   /pci@0,0/pci8086,8c18@1c,4/pci8086,1@0,1
5/dev/ppt3  8086   1521   /pci@0,0/pci8086,8c18@1c,4/pci8086,1@0,2
6/dev/ppt4  8086   1521   /pci@0,0/pci8086,8c18@1c,4/pci8086,1@0,3

Now these devices are ready to be passed through to bhyve

Create the ZFS dataset to store the bootdisk for the guest

This is the ZFS dataset that will be the storage device for the OpenBSD bhyve zone. Create the ZFS dataset as follows:

1omnios# zfs create -V 10G data/openbsd

Configure the OpenBSD bhyve zone

Zones are configured on OmniOS via the zonecfg command, this is also true for bhyve zones.

The following can be pasted into zonecfg once it has been executed. You will need to change any directives that are specific to your system.

 1# zonecfg -z openbsd
 2openbsd: No such zone configured
 3Use 'create' to begin configuring a new zone.
 4
 5create -b
 6set brand=bhyve
 7add attr
 8    set name=type
 9    set type=string
10    set value=openbsd
11end
12set zonepath=/zones/openbsd
13set ip-type=exclusive
14add net
15    set allowed-address=192.168.1.67/24
16    set physical=openbsd0
17end
18add device
19    set match=/dev/lofi/1
20end
21add attr
22    set name=disk
23    set type=string
24    set value=/dev/lofi/1
25end
26add device
27    set match=/dev/zvol/rdsk/data/openbsd
28end
29add attr
30    set name=bootdisk
31    set type=string
32    set value=/dev/zvol/rdsk/data/openbsd
33end
34add attr
35    set name=acpi
36    set type=string
37    set value=off
38end
39add attr
40    set name=bootrom
41    set type=string
42    set value=BHYVE_RELEASE
43end
44add device
45    set match=/dev/ppt1
46end
47add attr
48  set name=extra
49  set type=string
50  set value="-S -s 8:0,passthru,/dev/ppt1"
51end
52verify
53commit
54exit

The above is quite intuitive and should not need any explanation.

Install the OpenBSD bhyve zone

Once a new zone has been configured with zonecfg it needs to be installed. This is achieved with the following command:

1# zoneadm -z openbsd install

Boot the OpenBSD bhyve zone

Everything is now set and we can boot the OpenBSD bhyve zone as follows:

1# zoneadm -z openbsd boot

You now have 5 seconds to access the OpenBSD boot loader. After this time, you will not be able to access the OpenBSD bhyve zone via the console. Or at all... that bird will have flown and you will have to reboot the zone.

Enter the console on the OpenBSD bhyve zone

Configuration of the OpenBSD guest needs to be done via the console, as follows:

1# zlogin -C openbsd
2[Connected to zone 'openbsd' console]

Immeadiatley after entering the console, press the space bar & then the enter key to stop OpenBSD booting into the VGA Output mode.

Setting the console in the OpenBSD bhyve zone

After successfully accessing the console, the boot> prompt will appear. The OpenBSD guest now needs to be configured to redirect all output to the com0 serial port.

1boot> set tty com0

Boot the OpenBSD bhyve zone

Once the com0 serial console is set, the OpenBSD guest can be booted:

1boot> boot

OpenBSD can now be installed as normal. The Pass-Thru Ethernet Device will show in the network configuration as a normal device and no special care needs to be taken.

Where it gets tricky is the Disk Creation section, which I cover next

Setting up the root disk with uEFI

For me, GPT is the sure-fire way to make sure uEFI is installed... all the time... every time. You will see a warning about uEFI & GPT, but from my experiences, this can safley be ignored. And besides, we will do a full check after installation just to make sure uEFI is where it is supposed to be.

So when you reach that part of the installation that deals with the disk, we want to be installing onto sd0 and the creation looks a little something like this:

 1Available disks are: sd0 sd1.
 2Which disk is the root disk? ('?' for details) [sd0] sd0
 3No valid MBR or GPT.
 4Use (W)hole disk MBR, whole disk (G)PT or (E)dit? [whole] G
 5An EFI/GPT disk may not boot. Proceed? [no] yes
 6Setting OpenBSD GPT partition to whole sd0...done.
 7The auto-allocated layout for sd0 is:
 8#                size           offset  fstype [fsize bsize   cpg]
 9  a:       1218736.0K             1024  4.2BSD   2048 16384     1 # /
10  b:        262144.0K          2438496    swap                    
11  c:      10485760.0K                0  unused                    
12  d:       3145728.0K          2962784  4.2BSD   2048 16384     1 # /usr
13  e:       2097152.0K          9254240  4.2BSD   2048 16384     1 # /home
14  i:           480.0K               64   MSDOS                    
15Use (A)uto layout, (E)dit auto layout, or create (C)ustom layout? [a]
16/dev/rsd0a: 1190.2MB in 2437472 sectors of 512 bytes
176 cylinder groups of 202.50MB, 12960 blocks, 25920 inodes each
18/dev/rsd0e: 2048.0MB in 4194304 sectors of 512 bytes
1911 cylinder groups of 202.50MB, 12960 blocks, 25920 inodes each
20/dev/rsd0d: 3072.0MB in 6291456 sectors of 512 bytes
2116 cylinder groups of 202.50MB, 12960 blocks, 25920 inodes each
22Available disks are: sd0.
23Which disk do you wish to initialize? (or 'done') [done] 
24/dev/sd0a (a3b707add095aa86.a) on /mnt type ffs (rw, asynchronous, local)
25/dev/sd0e (a3b707add095aa86.e) on /mnt/home type ffs (rw, asynchronous, local, nodev, nosuid)
26/dev/sd0d (a3b707add095aa86.d) on /mnt/usr type ffs (rw, asynchronous, local, nodev)

Continue on with installing the file sets via http and when that is all finished, we will make those checks for the uEFI boot loader.

Checking the root disk for uEFI

Exit out of the installer and into the shell and from here we can verify all the right uEFI bits are in place:

 1CONGRATULATIONS! Your OpenBSD install has been successfully completed!
 2
 3When you login to your new system the first time, please read your mail
 4using the 'mail' command.
 5
 6Exit to (S)hell, (H)alt or (R)eboot? [reboot] S
 7To boot the new system, enter 'reboot' at the command prompt.
 8
 9openbsd# fdisk sd0
10Disk: sd0       Usable LBA: 64 to 20971456 [20971520 Sectors]
11   #: type                                 [       start:         size ]
12------------------------------------------------------------------------
13   1: EFI Sys                              [          64:          960 ]
14   3: OpenBSD                              [        1024:     20970433 ]

To be absolutely certain, we can check inside the uEFI partiton to make sure the bootx64.efi file is available.

1openbsd# mount /dev/sd0i /mnt2                                                 
2openbsd# ls /mnt2/efi/boot/     
3bootia32.efi bootx64.efi
4openbsd# umount /mnt2

Looking good!

Reboot the system

We can issue the reboot command and our new OpenBSD bhyve zone will come up cleanly:

 1# reboot
 2
 3>> OpenBSD/amd64 BOOTX64 3.50
 4boot> 
 5booting hd0a:/bsd: 12879176+2753552+324816+0+704512 [813343+128+1022112+747816]=0x125cf68
 6entry point at 0x1001000
 7[ using 2584424 bytes of bsd ELF symbol table ]
 8Copyright (c) 1982, 1986, 1989, 1991, 1993
 9	The Regents of the University of California.  All rights reserved.
10Copyright (c) 1995-2020 OpenBSD. All rights reserved.  https://www.OpenBSD.org
11
12OpenBSD 6.7 (GENERIC) #179: Thu May  7 11:02:37 MDT 2020
13    deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
14real mem = 1056288768 (1007MB)
15...
16...
17...
18...

Escaping from the console

To escape from the console, use ~~. You will be brought back to the OmniOS system.

Post installation tasks

There is nothing necessary to be done, however a little tidying up is always good.

This should be done when you have the OpenBSD bhyve zone in a shut-off state.

1# zonecfg -z openbsd
2
3remove device match=/dev/lofi/1
4remove attr name=disk
5verify
6commit
7exit

Unmount the lofi block device:

1lofiadm -d /dev/lofi/1

Connecting to the installed OpenBSD bhyve zone

Console and optionally ssh should be available to connect to the zones:

1# zlogin -C openbsd

or ssh into the OpenBSD bhyve zone if you have configured sshd.

1ssh 192.168.1.67

Wrapping up

There are quite a few ways to configure zones, this has been one of them. Some other tips that you may find easier are as follows:

  • In zonecfg, you can use zone templates to reduce typing a bit, so create -t bhyve in this case. Then you don't need to set brand, or set ip-type.
  • In the net section, you can set global-nic which makes the vnic get created and deleted automatically with VM boot.
  • There is also the options to use zadm or zcage in assisting you with the setup of Zones on OmnioOS.

Thanks to Andy at citrus-it for these tips.

{% if site.disqus_shortname %} {% include disqus_comments.html %} {% endif %}