How to install Void Linux + ZFS + ZFSBootMenu (Native ZFS Encryption on SSD + LVM/LUKS2 on HDD)

2021-03-14

(last time edited: 2021-05-22)

tags: linux, zfs, zbm, encryption

ZFS

The Z File System (ZFS) was created by Matthew Ahrens and Jeff Bonwick in 2001. ZFS was designed to be a next generation file system for Sun Microsystems’ OpenSolaris. In 2008, ZFS was ported to FreeBSD. The same year a project was started to port ZFS to Linux. However, since ZFS is licensed under the Common Development and Distribution License, which is incompatible with the GNU General Public License, it cannot be included in the Linux kernel. To get around this problem, most Linux distros offer methods to install ZFS.

Void Linux offers a ZFS module that can be installed in the kernel using DKMS hooks. And instead of GRUB we use ZFSBootMenu which is a boot program in development made for Void in mind and it functions properly in other Linux distributions too.

ZFSBootMenu

ZFSBootMenu is a Dracut module that intends to provide Linux distributions with an experience similar to FreeBSD's bootloader. By taking advantage of ZFS features, it allows a user to have multiple "boot environments" (with different distros, for example), manipulate snapshots before booting, and, for the adventurous user, even bootstrap a system installation via zfs recv.

ZBM has been in development for a while now. It replaces GRUB and any other bootloader completely. It has too many features and works like a charm.

I will show you how to install a very simple Void-on-ZFS setup. Thanks to zdykstra, ahesford and ericonr for developing ZBM and being very active in libera's #zfsbootmenu.

I used this guide and made a few personal additions.

At the end of this guide you will have a setup like this:

Download

Download hrmpf rescue tool from here.

Burn

Check your connected storage devices.

# lsblk

Burn the live image.

# dd if=hrmpf*.iso of=/dev/sdX

Reboot

# reboot

Boot

Boot the USB and log in to root by pressing any key.

Update

Update your live image.

# xbps-install -Su xbps

Check drives

Check your connected devices.

# lsblk

Partition SSD drive

Partition your SSD using fdisk.

# fdisk /dev/sdX

Create a new GPT partition table.

Command (m for help): g

Created a new GPT disklabel with disk identifier XxXXXXXXXXXXXX

Create the first partition. This will be your EFI partition. Choose a considerable size. (+200M, +300M)

Command (m for help): n

Partition number (1-128, default 1): 1

First sector (2048-xxxxxxxxxxxx, default 2048): ENTER

Last sector, +-sectors or +/-size{K,M,G,T,P} (2048-xxxxxxxxx, default xxxxxxxxx): +350M

Created a new partition 1 of type 'xxxxxxxxx' and of size 350 MiB.

Create a second partition. This will be your root ZFS partition.

Command (m for help): n

Partition number (2-128, default 2): 2

First sector (xxxxxxxxxxxx-xxxxxxxxxxxxx, default xxxxxxxxxx): ENTER

Last sector, +/-sectors or +/-size{K,M,G,T,P} (xxxxxxxxx-xxxxxxxx, default xxxxxxxxx): ENTER

Created a new partition 2 of type 'xxxxx' and of size XXXXXX GiB.

Write changes and exit

Command (m for help): w

The partition table has been altered.
Calling iotcl() to re-read partition table.
Syncing disks.

Re-read the partition table so the Linux kernel see the changes.

# kpartx /dev/sdX

Partition HDD drive

Partition your HDD secondary hard drive using fdisk.

# fdisk /dev/sdX

Create a new GPT partition table.

Command (m for help): g

Created a new GPT disklabel with disk identifier XxXXXXXXXXXXXX

Create the only partition needed. It will only serve as a home directory.

Command (m for help): n

Partition number (1-128, default 1): 1

First sector (2048-xxxxxxxxxxxx, default 2048): ENTER

Last sector, +-sectors or +/-size{K,M,G,T,P} (2048-xxxxxxxxx, default xxxxxxxxx): ENTER

Created a new partition 1 of type 'xxxxxxxxx' and of size XXXXXXXXXX GiB.

Write changes and exit

Command (m for help): w

The partition table has been altered.
Calling iotcl() to re-read partition table.
Syncing disks.

Re-read the partition table so the Linux kernel see the changes.

# kpartx /dev/sdX

ZFS (SSD)

Generate a /etc/hostid file.

# zgenhostid

Create a file and store your password in it. Choose a strong password, use something like KeePassXC to store it.

We will use zroot as the default name for your Void-on-ZFS filesystem. You can choose whatever name you want.

# echo some_password_you_want > /etc/zfs/zroot.key

Remove all permissions. Root will only have access to the key.

# chmod 000 /etc/zfs/zroot.key

Create your only and unique ZFS pool. -m none means it won't mount automatically. zroot will be name of the pool.

# zpool create -m none -O encryption=on -O keylocation=file:///etc/zfs/zroot.key -O keyformat=passphrase zroot /dev/sdX2

Create a ZFS filesystem. It won't mount automatically.

# zfs create -o mountpoint=none zroot/ROOT

# zfs create -o mountpoint=/ -o canmount=noauto zroot/ROOT/void."$(date +%Y.%m.%d)"

Create a temporary directory for your new system and a ZFS directory inside it.

# mkdir -p /mnt/nusys/etc/zfs

Export the pool.

# zpool export zroot

Re-import the pool.

# zpool import -N -R /mnt/nusys zroot

Now you will be prompted to enter the password you have chosen before to cipher with ZFS.

# zfs load-key -L prompt zroot

Now completely mount the root filesystem. It will be mounted automatically in /mnt/nusys

# zfs mount zroot/ROOT/void."$(date +%Y.%m.%d)"

LVM on LUKS2 (HDD)

Activate LUKS in the HDD single partition.

# cryptsetup luksFormat --type luks2 /dev/sdX

Open the LUKS partition.

# cryptsetup open /dev/sdX hdd0

Now create logical volumes.

# pvcreate /dev/mapper/hdd0

# vgcreate hdd0 /dev/mapper/hdd0

# lvcreate -n home -l 100%FREE hdd0

Format the only volume we created.

# mkfs.ext4 -F /dev/mapper/hdd0-home

Installation

First get the tarball name and store it in a variable.

# TAR=$(curl -s https://alpha.de.repo.voidlinux.org/live/current/ | grep -oP '(?<=").*x86_64-ROOTFS.*(?=")')

Download the file.

# wget https://alpha.de.repo.voidlinux.org/live/current/"$TAR"

Extract it on the new system mountpoint.

# tar -xf "$TAR" -C /mnt/nusys

Copy important files and settings

# cp /etc/hostid /mnt/nusys/etc

# cp /etc/resolv.conf /mnt/nusys/etc

# cp /etc/zfs/zroot.key /mnt/nusys/etc/zfs

Mount Bind

These directories are mandatory to bindmount from one system to another. It makes processes and devices be visible from both kernels. It's an important step before chrooting.

# mount -t proc /proc /mnt/nusys/proc

# mount -t sysfs /sys /mnt/nusys/sys

# mount -o bind /dev /mnt/nusys/dev

Add system settings

# echo void > /mnt/nusys/etc/hostname

An example of a configured rc.conf file should look similar to this:

HOSTNAME="void"
TIMEZONE="America/Argentina/Salta"
KEYMAP="la-latin1"

Update the new system

First upgrade the package manager.

# xbps-install -r /mnt/nusys -Su xbps

Now do a system upgrade.

# xbps-install -r /mnt/nusys -Su

The ROOTFS tarball already comes with base-voidstrap installed. You only have to install the kernel, zfs, etc.

# xbps-install -r /mnt/nusys linux linux-headers zfs zfsbootmenu refind cryptsetup lvm2

Autounlock LUKS partition

Create a key with random characters. This is the key that root will read when the system initiates.

# dd bs=1024 count=7 status=progress if=/dev/urandom of=/mnt/nusys/root/hdd0.key

We already have a passphrase key to unlock LUKS. You can delete that key slot if you want for more security. But we will make use of another LUKS slot and add the generated key file.

# cryptsetup luksAddKey /dev/sdX1 /mnt/nusys/root/hdd0.key

Remove all permissions from the file so only root can read it.

# chmod 000 /mnt/nusys/root/hdd0.key

Add the LUKS partition in crypttab.

# echo hdd0 /dev/disk/by-uuid/"$(blkid -s UUID -t TYPE=crypto_LUKS -o value /dev/sdX1)" /root/hdd0.key > /mnt/nusys/etc/crypttab

Now tell Dracut to read crypttab. This is necessary for the initramfs generation.

# echo install_items+=\" /root/hdd0.key /etc/crypttab \" > /mnt/nusys/etc/dracut.conf.d/luks.conf

Dracut

If you take a look we are adding the ZFS module and the zroot.key plaintext we created before.

Create a zol.conf for Dracut.

# touch /mnt/nusys/etc/dracut.conf.d/zol.conf

And add the following text.

nofsck="yes"
add_dracutmodules+=" zfs "
omit_dracutmodules+=" btrfs rngd "
install_items+=" /etc/zfs/zroot.key "

ZFSBootMenu

Create a pool cachefile to discover pools quicker on boot.

# chroot /mnt/nusys zpool set cachefile=/etc/zfs/zpool.cache zroot

Set the default boot environment.

# chroot /mnt/nusys zpool set bootfs=zroot/ROOT/void."$(date +%Y.%m.%d)" zroot

Add command-line kernel arguments to be used when booting. I personally use net.ifnames=0 and biosdevname=0 to get consistent ethernet card names. Instead of ens0dpxxx or similar, it will show eth0. This is very nice to use in addition to ndhc.

# chroot /mnt/nusys zfs set org.zfsbootmenu:commandline="spl_hostid=$(hostid) ro quiet cgroup_no_v1=all net.ifnames=0 biosdevname=0" zroot/ROOT

Configure ZBM. The config.yaml is very intuitive and only works for the kernel image that boots ZBM.

Open the config.yaml file with your favorite text editor.

# vi /mnt/nusys/etc/zfsbootmenu/config.yaml

At the end it should look like this:

Global:
   ManageImages: true
   BootMountPoint: /boot/efi
   DracutConfDir: /etc/zfsbootmenu/dracut.conf.d
Components:
   ImageDir: /boot/efi/EFI/void
   Versions: 1
   Enabled: true
   syslinux:
     Config: /boot/syslinux/syslinux.cfg
     Enabled: false
EFI: 
   ImageDir: /boot/efi/EFI/void
   Versions: 1

Fstab

No need to specificy the root ZFS filesystem on fstab. Zpool mounts it already. We will only mount tmpfs and home.

This is an example of a Fstab configuration file. Direct file path is /mnt/nusys/etc/fstab.

Remember to replace the blkid information with yours. First UUID points to the boot/efi. Second UUID is the LUKS2 device.

tmpfs /tmp tmpfs defaults 0 0
UUID=$(blkid -s UUID -t TYPE=vfat -o value /dev/sdX1) /boot/efi vfat defaults 0 0
UUID=$(blkid -s UUID -o value /dev/mapper/hdd0-home) /home ext4 defaults 0 0

At the end of this guide we can remove the vfat partition from fstab. It's not necessary to mount it automatically all the time. It's only necessary to reconfigure ZBM and efibootmgr or rEFInd.

EFI

Format the efi partition we have made at the beginning.

# mkfs.vfat /dev/sdX1

Create a /boot/efi directory in the new system.

# chroot /mnt/nusys mkdir /boot/efi

Mount that directory. By using this command it will already assume the device by reading fstab, so there is no need to declare it.

# chroot /mnt/nusys mount /boot/efi

Reconfigure

Reconfiguring the kernel package linuxX.XX executes post-installation kernel hooks: 20-dracut, 50-efibootmgr and 50-refind. You can find these hooks in the /etc/kernel.d directory. Also the installation of the kernel package + headers will populate /boot with System.map-x.xx.x_x, config-x.xx.x_x, vmlinuz-x.xx.x_x files.

# xbps-reconfigure -f /mnt/nusys -f linuxX.XX

Reconfiguring the package ZFS uses DKMS to build and install the ZFS module for each kernel version installed in your system. Also creates the initramfs-x.xx.x_x.img files specific to each kernel that go in /boot directory.

# xbps-reconfigure -r /mnt/nusys -f zfs

Reconfiguring ZFSBootMenu uses the binary generate-zbm included in the zfsbootmenu pkg, to generate the ZFSBootMenu initramfs image with an specific configuration we set up before in /mnt/nusys/etc/zfsbootmenu/config.yaml. This command basically creates a self-contained Linux system that will show an ncurses boot menu in similar fashion to FreeBSD where you can do many things, such as choosing different kernel versions, adding command line kernel options, etc. This self-contained system is embedded inside these files: initramfs-x.xx.x_x.img and vmlinuz-x.xx.x_x. These files are placed in /boot/efi/EFI/void. There is no need to generate-zbm multiple times. Otherwise the /boot/efi/EFI/void will be full of different ZBM kernel images and there will no more space in the partition.

# xbps-reconfigure -r /mnt/nusys -f zfsbootmenu

Anyways I just prefer doing...

# xbps-reconfigure -fa

since reconfiguring all the system is the way to go.

Reconfiguring everything also helps fixing any tarball extraction misconfiguration.

If you wanna make sure your initramfs image contains the ZFS module, run this command:

# lsinitrd /boot/initramfs-some-kernel-version.img | grep zfs.ko

and this command too

# dkms status

If by some reason XBPS wasn't able to unpack the kernel files and headers in /boot; zfsbootmenu and generate-zbm may get errors like:

Unable to find latest kernel; specify version or path manually.

Force a reinstallation on the kernel and reconfigure everything again.

# xbps-install -f linuxX.XX linuxX.XX-headers

Option 1. Boot using efibootmgr + gummiboot directly to ZBM (no rEFInd)

This option 1 is is based on a guide wrote by ahesford. You can look at it here.

Install efibootmgr and gummiboot.

Edit ZBM's config.yaml.

# vi /mnt/nusys/etc/zfsbootmenu/config.yaml

Remember to replace $HOSTID with your ZFS hostid.

# echo $(hostid)

At the end it should look like this.

Global:
  ManageImages: true
  BootMountPoint: /boot/efi
  DracutConfDir: /etc/zfsbootmenu/dracut.conf.d

Components:
  ImageDir: /boot/efi/EFI/void
  Versions: 1
  Enabled: true
  syslinux:
    Config: /boot/syslinux/syslinux.cfg
    Versions: 1

EFI:
  ImageDir: /boot/efi/EFI/void
  Versions: false
  Enabled: true

Kernel:
  CommandLine: ro quiet loglevel=0 spl_hostid=$HOSTID zbm.prefer=zroot
  Prefix: ZBM

Now remove all initramfs and vmlinuz previously generated with xbps-reconfigure.

# rm /mnt/nusys/boot/efi/EFI/void/*

And re-generate initramfs vmlinuz and an .EFI file.

# xbps-reconfigure -f zfsbootmenu

If you look at the directory.

# ls /mnt/nusys/boot/efi/EFI/void

you will see there are 3 files now.

initramfs-x.x.x_x.img
zbm-x.x.x_x
ZBM.EFI

Now add an efibootmgr entry.

# efibootmgr -c -d /dev/sdX -p 1 -L "ZFSBootMenu" -l \\EFI\\VOID\\ZBM.EFI

You can look at all EFI entries.

# efibootmgr -v

And delete all entries you don't want... for example:

# efibootmgr -b 0004 -B

Option 2. Boot to ZBM using rEFInd

Install rEFInd.

rEFInd is a very good looking an interactive UI menu that appears after the BIOS. It can make dual boot a lot whole easier and fun. While rEFInd is great, it is more lines of codes than a simple efibootmgr entry and configuring it can be a little pain in the ass.

blogimg

# chroot /mnt/nusys refind-install

Remove the default configuration.

# rm /mnt/nusys/boot/refind_linux.conf

Add the following settings to a new rEFInd configuration.

20 seconds for timeout is a nice time. On boot time you will have time to either load the kernel directly or press the ESC key that opens the ZBM menu.

0 is you want to boot directly without being prompt for any user interaction.

To configure our ZBM with rEFInd we need to create a configuration file.

First create a refind_linux.conf file.

# touch /mnt/nusys/boot/efi/EFI/void/refind_linux.conf

This is an example of a refind_linux.conf file.

Remember to replace HOST_ID with your ZFS hostid.

# echo $(hostid)

"Boot default" "zfsbootmenu:POOL=zroot spl_hostid=HOST_ID zbm.timeout=20 ro quiet loglevel=0"
"Boot to menu" "zfsbootmenu:POOL=zroot spl_hostid=HOST_ID zbm.timeout=-1 ro quiet loglevel=0"

This is a visual example of what you can do in the ZBM menu.

blogimg

If you want to load and debug a kernel using ZBM you can remove quiet and add loglevel=7.

Unmount

# umount /mnt/nusys/dev/pts

# umount /mnt/nusys/dev

# umount /mnt/nusys/sys

# umount /mnt/nusys/proc

# umount /mnt/nusys/boot/efi

# zpool export zroot

That's pretty much it! Enjoy your new Void-on-ZFS system.


Hrmpf

hrmpf is a custom Void Linux image made with void-mklive made by Leah from the Void community. It functions as a rescue system because of the many tools already integrated in it. Loads ZFS and lots of other modules. This is the Void distribution by excellence if you wanna experiment, solve problems, install new systems, etc. Think of it as a swiss knife. You can download the image here.

Some features:

If you find yourself in trouble trying to fix your new Void-on-ZFS system, boot into hrmpf and use similar commands to this:

Warning: Do not run these commands blindly, think of what you are doing, it's a mere example of the workflow.

# mkdir /mnt/rescue

# zpool import -N -R /mnt/rescue zroot

# zfs load-key -L prompt zroot

# zfs mount zroot/ROOT/void.xxxxxxx

# zfs mount zroot/home

# mount -t proc /proc /mnt/rescue/proc

# mount -t sysfs /sys /mnt/rescue/sys

# mount -o bind /dev /mnt/rescue/dev

# mount /dev/sdX1 /mnt/rescue/boot/efi

# xbps-reconfigure -r /mnt/rescue -f zfs linuxXX.X zfsbootmenu

# chroot /mnt/rescue refind-install

# umount /mnt/rescue/proc

# umount /mnt/rescue/sys

# umount /mnt/rescue/dev

# zpool export zroot

Or you can simple boot into ZBM recovery mode and do:

# mkdir -p /mnt/rescue

# zpool import -N -R /mnt/rescue zroot

# zfs load-key -L prompt zroot

# zfs list

# zfs mount zroot/ROOT/void.xxxxxx

# chroot /mnt/rescue

Then do whatever you want and reboot.