Installing CentOS 7 on FreeNAS

Note This is a mirror of Vincent Danen's blog.

For the purposes of this post/tutorial, you need to be running FreeNAS 9.10 (you can probably do this easily enough with FreeBSD, but I've not tried). There is also documentation on Using iohyve.

From your FreeNAS system you need to know your ethernet interface name (in the web UI go to Network -> Network Interfaces, in my case em0) and the storage pool name (Storage -> Volumes, in my case the pool is named storage). The actual setup of iohyve needs to be done as root over SSH, so you'll need that running as well.

As root, we need to create the environment iohyve requires. I used the following commands to create the pool for its use:

# iohyve setup pool=storage kmod=1 net=em0
Setting up iohyve pool...
On FreeNAS installation.
Checking for symbolic link to /iohyve from /mnt/iohyve...
Symbolic link to /iohyve from /mnt/iohyve successfully created.
Loading kernel modules...
bridge0 is already enabled on this machine...
Setting up correct sysctl value...
net.link.tap.up_on_open: 0 -> 1

This tells iohyve to install the required ZFS datasets and kernel modules. We use kmod=1 to tell iohyve to load the required kernel module, pool=storage tells it which pool to use for files (in this case, storage) and net=em0 sets up the network bridge to this interface (iohyve can only be bound to a single interface). You can use multiple pools for iohyve, however I only have one pool on the system.

Next, you need to create a few tunables in FreeNAS. Heading back to the web UI, go to System -> Tunables and create the following two tunables:

• variable: iohyve_enable, values: YES, type: rc_conf
• variable: iohve_flags, values: kmod=1 net=em0, type: rc_conf

The iohyve_enable variable tells FreeNAS to load iohyve support at boot, and the iohyve_flags are the same kmod and net options we used when setting up iohyve initially.

The next step is to download an ISO image for iohyve to use for installing a virtual machine. In my case, I want to run CentOS 7. There are plenty of mirrors to choose from for the minimal ISO which is probably what you want since you can install any specific software required using yum after it's installed.

# iohyve fetch http://centos.mirror.iweb.ca/7/isos/x86_64/CentOS-7-x86_64-Minimal-1611.iso
Fetching http://centos.mirror.iweb.ca/7/isos/x86_64/CentOS-7-x86_64-Minimal-1611.iso...
/iohyve/ISO/CentOS-7-x86_64-Minimal-1611.iso/C100% of  680 MB    9 MBps 01m08s

You need to use iohyve to fetch the ISO image (somewhat annoyingly) so you can't just copy an existing ISO over (although I believe you can do a fetch over NFS or provide it locally via HTTP from another system on your network).

Once you have the ISO downloaded, you can configure a new virtual machine. Check to make sure you have the ISO available:

# iohyve isolist
Listing ISO's...
CentOS-7-x86_64-Minimal-1611.iso

Then, to create the machine with 20GB of space (the same size as my existing KVM machine for IPA, which is more than enough):

# iohyve create ipa-slave 20G
Creating ipa-slave...
# iohyve list
Guest      VMM?  Running  rcboot?  Description
ipa-slave  NO    NO       NO       Sun Mar 12 16:39:10 MDT 2017

Now you can configure the specifics of the machine:

# iohyve set ipa-slave ram=1G cpu=1 os=custom loader=grub-bhyve
Setting ipa-slave ram=1G...
Setting ipa-slave cpu=1...
Setting ipa-slave os=cuscom...
Setting ipa-slave loader=grub-bhyve...

This sets my virtual machine to have 1GB of RAM, use one virtual CPU, use the "custom" operating system type (we need this later, even though we will be using CentOS 7), and uses the grub-bhyve loader which is required by Linux guests. The iohyve wiki has more details on operating system types and which values to use depending on which Linux operating system you intend to install.

When using a CentOS 7 guest, iohyve currently cannot boot from an XFS partition (which is the default), and due to the limitations of the commandline installer, we can't tell Anaconda to use something other than XFS. Another thing I found, with some trial and error, is you want to use traditional partitions and not the LVM-based partition scheme (so plan out your filesystem in advance to ensure you have enough size!). This is the main reason for using the "custom" operating system type. We'll fix that later.

To work around this, we'll use a simple kickstart file to get us to a minimal working system from which we can install the rest of what we want.

In order to make grub boot and use the kickstart file, you need to edit /iohyve/ipa-slave/grub.cfg so it looks like:

linux (cd0)/isolinux/vmlinuz inst.ks=http://somewhere.internal/ks.cfg
initrd (cd0)/isolinux/initrd.img
boot

and the ks.cfg file would look something like (see the documentation for more info):

#version=RHEL7
# System authorization information
auth --enableshadow --passalgo=sha512

# Use CDROM installation media
cdrom
# Use text install
text
# Run the Setup Agent on first boot
firstboot --enable
ignoredisk --only-use=sda
# Keyboard layouts
keyboard --vckeymap=us --xlayouts='us'
# System language
lang en_US.UTF-8

# Network information
network  --bootproto=static --device=eth0 --gateway=192.168.1.1 --ip=192.168.1.16 --nameserver=192.168.1.1 --netmask=255.255.255.0 --noipv6 --activate
network  --hostname=ipa-slave.mydomain.com
# Root password
rootpw --iscrypted $6$/GdEAa2DwhlmU.Vr$R/L.fEc6QwtFiTMLd04HR1SuS7NrsdA.NuQyQ17RbBk8p37oGD/hVvRIOw0v5x6pSC6uU4NigueNmEXvQ8pzo0
# System services
services --enabled="chronyd"
# System timezone
timezone America/Edmonton --isUtc --ntpservers=ntp.mydomain.com
# System bootloader configuration
bootloader --location=mbr --boot-drive=sda
# Partition clearing information
clearpart --drives=sda --all
# Disk partitioning information
autopart --type=plain --fstype="ext4"

%packages
chrony
%end

%addon com_redhat_kdump --disable --reserve-mb='auto'

%end

Before starting the installation, make sure you can retrieve that file with something like curl. Next, start the installation:

# iohyve isolist
Listing ISO's...
CentOS-7-x86_64-Minimal-1611.iso
# iohyve install ipa-slave CentOS-7-x86_64-Minimal-1611.iso
Installing ipa-slave...
GRUB Process does not run in background....
If your terminal appears to be hanging, check iohyve console ipa-slave in second terminal to complete GRUB process...

From another terminal, ssh into your FreeNAS server again in order to connect to the serial console by using:

# iohyve console ipa-slave
Starting console on ipa-slave...
~~. to escape console [uses cu(1) for console]
Connected
...
Starting installer, one moment...
anaconda 21.48.22.93-1 for CentOS Linux 7 started.
 * installation log files are stored in /tmp during the installation
 * shell is available on TTY2
 * when reporting a bug add logs from /tmp as separate text/plain attachments
14:21:39 Not asking for VNC because of an automated install
14:21:39 Not asking for VNC because text mode was explicitly asked for in kickstart

Starting automated install..
...

Now sit back while it automatically installs. This will be quite the minimal install, however it will get you up and running with an ext4-based system that iohyve can boot up, and from there you can install individual packages or package groups.

When the install is complete, you will see something like this on the console:

[  OK  ] Started Restore /run/initramfs.
[  OK  ] Reached target Shutdown.
dracut Warning: Killing all remaining processes
Rebooting.
[  819.859124] Restarting system.

However, it's not actually restarting the system. Switch back to the other console and you will see:

Unhandled ps2 keyboard command 0xf6

# iohyve list
Guest      VMM?  Running  rcboot?  Description
ipa-slave  YES   NO       NO       Sun Mar 12 16:39:10 MDT 2017

As you can see from the list command, the virtual machine is not running even though it said it was restarting. In order to start the machine, you must use:

# iohyve start ipa-slave
Starting ipa-slave... (Takes 15 seconds for FreeBSD guests)
[root@heimdall] ~# GRUB Process does not run in background....
If your terminal appears to be hanging, check iohyve console ipa-slave in second terminal to complete GRUB process...

Now switch back to the original console (that you've not disconnected) and you will see the system booting. I had some difficulty with dracut-initqueue warning about incessant timeouts, so it took some time for the system to boot and even then I ended up in a rescue shell.

As annoying as this is, it's not too terribly difficult to solve although I hate how hackish it needs to be. The root of the problem seems to be that dracut wants to connect to this kickstart file before the network is up. So we need to do some chroot shenanigans and fix the initramfs:

# mkdir /mnt
# mount /dev/sda3 /mnt
# mount /dev/sda1 /mnt/boot
# chroot /mnt
# cd /boot
# cp initramfs-3.10.0-514.el7.x86_64.img initramfs-3.10.0-514.el7.x86_64.img.bak
# dracut -f /boot/initramfs-3.10.0-514.el7.x86_64.img 3.10.0-514.el7.x86_64
# ls -al initramfs*
-rw-------  1 root root 45180381 Mar 14 08:24 initramfs-0-rescue-715baff9360a47a89af2ddbc55b9f0cf.img
-rw-------  1 root root 44766225 Mar 14 09:26 initramfs-3.10.0-514.el7.x86_64.img
-rw-------  1 root root 17437682 Mar 14 09:25 initramfs-3.10.0-514.el7.x86_64.img.bak

Judging by the difference in size between the rescue image, the newly created image, and the previous one that was copied, this is a pretty strong indication that something is missing.

Also, there are a few more steps to do before we can get CentOS 7 booted properly. The first is to edit /iohyve/ipa-slave/grub.conf and remove the kickstart reference. I'm not 100% sure this is required after we perform the next action, but after a lot of beard-tugging (and to spare you the same) I'm suggesting you remove it. Also, you need to set the operating system type back to CentOS 7:

# iohyve set ipa-slave os=centos7
Setting ipa-slave os=centos7...

If you've stopped the virtual machine, great, if it's still waiting for network timeouts, you can forcibly stop it with iohyve destroy ipa-slave and then we can use iohyve start ipa-slave to start it back up again (this from the session not attached to the console, of course).

Now on the console, you should be able to watch the virtual machine boot and arrive at a login prompt.

Once you login, you probably want to install a few other things given we opted for the minimal install:

# yum update -y
# yum install net-tools vim-enhanced zsh ipa-server
# systemctl status sshd

The last is to make sure that sshd is running so you can ssh in and carry on (at least for me, an 80x25 console is pretty darn tiny). I also prefer the enhanced vim, and having tools like ipaddr and ifconfig are just plain old handy, and of course the whole point of this exercise was to set this up as an IPA server.

Finally, once you verify you can ssh into the server, disconnect from the console by typing the tilde and CTRL-D (so ~ + CTRL-D).

To finish up, let's give the virtual machine a decent description and tell it to start at boot:

# iohyve set ipa-slave description="IPA Slave server"
Setting ipa-slave description=IPA Slave server...
#  iohyve set ipa-slave boot=1
Setting ipa-slave boot=1...
# iohyve list
Guest      VMM?  Running  rcboot?  Description
ipa-slave  YES   YES      YES      IPA Slave server

At this point, you can make a backup or snapshot of the virtual machine (which is one of the first things I wanted to do after all of the effort of figuring out the above!):

# iohyve snap ipa-slave@base-install-20170314
Taking snapshot ipa-slave@base-install-20170314
# iohyve snaplist
ipa-slave@base-install-20170314

Currently there is not a way to remove snapshots.