Softraid and crypto for OpenBSD 5.3/5.4

DISCLAIMER: This how-to must be taken as is, it should not replace the official documentation and is not meant to do so. It may be useful as these features are quite new and not heavily documented on the net.

OpenBSD supports booting from a raid volume since version 5.3. Before that, the way to have redundancy for the root partition was to place /altroot on a second disk and to manually switch to it in the case of failure of the first disk.

It also supports booting from encrypted volumes. Sadly it doesn’t supports booting from raid+encrypted volumes yet (they’re working on it).

For my setup I want to have redundancy (meaning RAID1) and, ideally, crypto everywhere. As it’s not yet possible I decided to create one softraid1 partition containing “/” and a second softraid1/encrypted partition containing some mount points: /tmp, /var, /usr, /usr/X11R6, /usr/local, /usr/src, /usr/obj

Below, I’ll describe the steps to obtain such setup.

Setup part:

1. boot an install media (I used PXE here)

2. drop to a (S)hell in the installation program

3. create the devices nodes:

cd /dev
sh MAKEDEV sd0   # <- our 1st HDD
sh MAKEDEV sd1   # <- our 2nd HDD
sh MAKEDEV sd2   # <- our 1st RAID1 volume
sh MAKEDEV sd3   # <- our 2nd RAID1 volume
sh MAKEDEV sd4   # <- our 2nd RAID1 volume + crypto

4. initialize the MBR (i = initialize, y = answer yes):

fdisk -iy sd0
fdisk -iy sd1

5. partition the disks. We will create one “a” partition for the raid1 root fs, “b” for the encrypted one and “d” for the raid1 + crypto

disklabel -E sd0
a a   #1G, FS type: RAID root partition
a b   #2G, swap partition (OpenBSD automatically encrypts swap)
a d   #[all available space], FS type: RAID # RAID partition, which will contain all other partitions
w     #write
q     #quit

6. do the same for sd1:

disklabel -E sd1

7. create two RAID1 devices, one for the / and one for the encrypted partition:

bioctl -c 1 -l /dev/sd0a,/dev/sd1a softraid0
bioctl -c 1 -l /dev/sd0d,/dev/sd1d softraid0

8. two devices should be created: sd2 and sd3, we will empty their first few sectors:

dd if=/dev/zero of=/dev/rsd2c bs=1m count=1
dd if=/dev/zero of=/dev/rsd3c bs=1m count=1

9. create a partition on the new devices:

disklabel -E sd2
a a   #whole disk, FS type BSD
w     #write
q     #quit

10. and for the crypto partition:

disklabel -E sd3
a a   #whole disk, FS type RAID
w     #write
q     #quit

11. create the crypto partition:

bioctl -c C -r 8192 -l /dev/sd3a softraid0

12. start the installer:

install

13. choose to use (W)hole disk sd2 and partition it like that:

a a   #whole disk, FS BSD mount /
w     #write
q     #quit

14. Which one do you want to initialize? sd4

a d   #size 4G, /tmp
a e   #size 7G, /var
a f   #size 2G, /usr
a g   #size 1G, /usr/X11R6
a h   #size 7G, /usr/local
a i   #size 2G, /usr/src
a j   #size 2G, /usr/obj
a k   #size [left space ~24G], /home
w     #write
q     #quit

15. choose your mirror and the stuffs you want to install

16. finish the install and reboot

OS part

the boot should yell that some partitions can’t be mounted and drop you to a shell. It’s normal, crypted partitions aren’t supported out of the box by the boot process. You will have to decrypt the partition by hand (sd3a), then you will be asked for your passphrase:

bioctl -c C -l /dev/sd3a softraid0 && exit

next step we will do a modification of the boot order asking for the passphrase of the encrypted device instead of “crashing” with error messages. We will add the following script to the end of the file /etc/rc.conf.local:

bioctl sd4 > /dev/null 2>&1

if [ $? -ne 0 ]; then
    echo unlocking encrypted device
bioctl -c C -l /dev/sd3a softraid0
  fi

This script will hang the boot process until the passphrase for the encrypted device is entered, then it will be able to mount the system partitions that are on the crypted sd4 device (/tmp, /usr, …).

The location of my script is a bit tricky. In theory I should have put it in /etc/rc.securelevel, the problem is that rc.securelevel is called too late in the boot process, after the mount of the partitions.

17. reboot and check if the passphrase is asked and working, if not redo the step “17”.

Next step is only required for OpenBSD 5.3, it has been corrected since

The last “problem” we have is when we shutdown your system, OpenBSD will remove the devices approximately in the same order they are created. “sd3” gets shutdown before sd4, hence sd4 will be unable to write metadata to the underlying sd3 device. The next reboot will say that the sd4 device was not correctly unmounted and may request a fsck. To avoid this we will create a script that unmount the partitions located on the crypted device and then destroy the crypted disk before the sd3 devices gets removed.

18. add the following to the /etc/rc.shutdown file:

for device in $mounted; do
bioctl $device | grep -q CRYPTO
iscrypto="$?"

if [ "$iscrypto" -eq 0 ]; then
echo $device is a crypto device, umounting its partitions and delete crypto volume
# umount partitions related to crypto device in reverse order of mount output
tounmount=`mount | grep "^/dev/$device" | cut -f 1 -d ' ' | awk '{print FNR, $0}' | sort -nr | cut -f 2 -d ' '`

for cryptdev in $tounmount; do
umount -f $cryptdev
    done

bioctl -d $device

  fi
done

Last but not least, if you have some sensitive files containing passwords like IPsec secrets or pf configurations it may be good to put them into an encrypted partition, like /home, and symlink them to their official location.

Thanks for reading!

Romain

5 comments

  1. I was reading the change log for 5.4 [http://www.openbsd.org/plus54.html] and I noticed the line that reads “softraid(4) disciplines are now shutdown in reverse attach order, making manually stacked volumes more practical.” If I am using this blog post as a guide, but I am using OpenBSD 5.4 instead of 5.3, does this mean I might not have to edit /etc/rc.shutdown as in step 18?

    Thanks for the writing this blog post, “Softraid and crypto for OpenBSD…” is exactly what I’m trying to accomplish.

  2. You are right I just checked my setup against openbsd 5.4, the crypto volume is un-mounted first.

    The hack is not relevant anymore, I’ll update my blogpost.

    Thanks for pointing this out.

  3. I believe there is a non-critical error in the script that is to be appended to the end of /etc/rc.conf.local:

    2&>1 (incorrect)
    2>&1 (correction)

    I made this determination while reading about “command > /dev/null 2>&1” on the following web page: [http://www.xaprb.com/blog/2006/06/06/what-does-devnull-21-mean/]

  4. could follow his guide is a success

    I have just 2 problems

    Number 1 – I have 2 HD 3T each step to follow me disappeared 700G
    Do you know why?

    # df -h
    Filesystem Size Used Avail Capacity Mounted on
    /dev/sd2a 1001M 57.6M 893M 6% /
    /dev/sd4k 982G 8.0K 933G 0% /home
    /dev/sd4d 4.9G 8.0K 4.7G 0% /tmp
    /dev/sd4f 4.9G 413M 4.3G 9% /usr
    /dev/sd4g 4.9G 193M 4.5G 4% /usr/X11R6
    /dev/sd4h 9.8G 214K 9.3G 0% /usr/local
    /dev/sd4j 4.9G 2.0K 4.7G 0% /usr/obj
    /dev/sd4i 4.9G 2.0K 4.7G 0% /usr/src
    /dev/sd4e 1020G 7.4M 969G 0% /var

    Number 2 – When you reboot, it asks me the passphrase (while I can not login via ssh), it is easy to write when you are physically in front of the machine.
    But my remote server, you can only connect to KVM / console and it costs me money when I order.

    Normally no reboot both, but sometimes necessary.

    Do you could do with ssh login before, and so enter the passphrase in ssh?

    any ideas are welcome

    thanks for everything

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s