Andrew's blog

I've spent the last couple of days trying to deploy Fedora CoreOS to some physical hardware/bare metal for a colleague using the official PXE installer from Fedora CoreOS. It wasn't very pleasant, and just wouldn't work reliably.

Maybe my expectations were to high, in that I thought I could use Ignition to prepare more of the system for me, as my colleague has been able to bare metal installs correctly. I just tried to use Ignition as documented.

A few interesting aspects I encountered:

1. The PXE installer for it has a 618MB initrd file. This takes quite a while to transfer via tftp!
2. It can't build software RAID for the main install device (and the developers have no intention of adding this), and it seems very finicky to build other RAID sets for other partitions.
3. And, well, I just kept having problems where the built systems would hang during boot for no obvious reason.
4. The time to do an installation was incredibly long.
5. The initrd image is really just running coreos-installer against the nominated device.

During the night I got feed up with that process and wrote a Fully Automatic Installer (FAI) profile that'd install CoreOS instead. I can now use setup-storage from FAI using it's standard disk_config files. This allows me to build complicated disk configurations with software RAID and LVM easily.

A big bonus is that a rebuild is a lot faster, timed from typing reboot to a fresh login prompt is 10 minutes - and this is on physical hardware so includes BIOS POST and RAID controller set up, twice each.

I thought this might be of interest to other people, so the FAI profile I developed for this is located here: https://github.com/catalyst-cloud/fai-profile-fedora-coreos

FAI was initially developed to deploy Debian systems, it has since been extended to be able to install a number of other operating systems, however I think this is a good example of how easy it is to deploy non-Debian derived operating systems using FAI without having to modify FAI itself.

I found to make all this work I had to piece together a bunch of information from different locations. This fills in some of the blanks from the official Raspberry Pi documentation. See here, here, and here.

Image

Download the latest raspbian image from https://www.raspberrypi.org/downloads/raspbian/ and unzip it. I used the lite version as I'll install only what I need later.

To extract the files from the image we need to jump through some hoops. Inside the image are two partitions, we need data from each one.

 # Make it easier to re-use these instructions by using a variable
 IMG=2017-04-10-raspbian-jessie-lite.img
 fdisk -l $IMG

You should see some output like:

 Disk 2017-04-10-raspbian-jessie-lite.img: 1.2 GiB, 1297862656 bytes, 2534888 sectors
 Units: sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytes
 Disklabel type: dos
 Disk identifier: 0x84fa8189
 
 Device                               Boot Start     End Sectors  Size Id Type
 2017-04-10-raspbian-jessie-lite.img1       8192   92159   83968   41M  c W95 FAT32 (LBA)
 2017-04-10-raspbian-jessie-lite.img2      92160 2534887 2442728  1.2G 83 Linux

You need to be able to mount both the boot and the root partitions. Do this by tracking the offset of each one and multiplying it by the sector size, which is given on the line saying "Sector size" (typically 512 bytes), for example with the 2017-04-01 image, boot has an offset of 8192, so I mount it like this (it is VFAT):

 mount -v -o offset=$((8192 * 512)) -t vfat $IMG /mnt
 # I then copy the data off:
 mkdir -p /data/diskless/raspbian-lite-base-boot/
 rsync -xa /mnt/ /data/diskless/raspbian-lite-base-boot/
 # unmount the partition now:
 umount /mnt

Then we do the same for the root partition:

 mount -v -o offset=$((92160 * 512)) -t ext4 $IMG /mnt
 # copy the data off:
 mkdir -p /data/diskless/raspbian-lite-base-root/
 rsync -xa /mnt/ /data/diskless/raspbian-lite-base-root/
 # umount the partition now:
 umount /mnt

DHCP

When I first set this up, I used OpenWRT on my router, and I had to patch /etc/init/dnsmasq to support setting DHCP option 43. As of the writting of this article, a similar patch has been merged, but isn't in a release yet, and, well, there may never be another release of OpenWRT. I'm now running LEDE, and the the good news is it already has the patch merged (hurrah!). If you're still on OpenWRT, then here's the patch you'll need:

https://git.lede-project.org/?p=source.git;a=commit;h=9412fc294995ae2543fabf84d2ce39a80bfb3bd6

This lets you put the following in /etc/config/dnsmasq, this says that any device that uses DHCP and has a MAC issued by the Raspberry PI Foundation, should have option 66 (boot server) and option 43 set as specified. Set the IP address on option 66 to the device that should be used for tftp on your network, if it's the same device that provides DHCP then it isn't required. I had to set the boot server, as my other network boot devices are using a different server (with an older tftpd-hpa, I explain the problem further down).

 config mac 'rasperrypi'
         option mac 'b8:27:eb:*:*:*'
         option networkid 'rasperrypi'
         list dhcp_option '66,10.1.0.253'
         list dhcp_option '43,Raspberry Pi Boot'

tftp

Initially I used a version of tftpd that was too old and didn't support how the RPi tried to discover if it should use the serial number based naming scheme. The version of tftpd-hpa Debian Jessie works just fine. To find out the serial number you'll probably need to increase the logging of tftpd-hpa, do so by editing /etc/default/tftpd-hpa and adding "-v" to the TFTP_OPTIONS option. It can also be useful to watch tcpdump to see the requests and responses, for example (10.1.0.203 is the IP of the RPi I'm working with):

  tcpdump -n -i eth0 host 10.1.0.203 and dst port 69

This was able to tell me the serial number of my RPi, so I made a directory in my tftpboot directory with the same serial number and copied all the boot files into there. I then found that I had to remove the init= portion from the cmdline.txt file I'm using. To ease debugging I also removed quiet. So, my current cmdline.txt contains (newlines entered for clarity, but the file has it all on one line):

idwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs
nfsroot=10.1.0.253:/data/diskless/raspbian-lite-base-root,vers=3,rsize=1462,wsize=1462
ip=dhcp elevator=deadline rootwait hostname=rpi.etc.gen.nz

NFS root

You'll need to export the directories you created via NFS. My exports file has these lines:

/data/diskless/raspbian-lite-base-root	10.1.0.0/24(rw,no_root_squash,sync,no_subtree_check)
/data/diskless/raspbian-lite-base-boot	10.1.0.0/24(rw,no_root_squash,sync,no_subtree_check)

And you'll also want to make sure you're mounting those correctly during boot, so I have in /data/diskless/raspbian-lite-base-root/etc/fstab the following lines:

10.1.0.253:/data/diskless/raspbian-lite-base-root   /       nfs   rw,vers=3       0   0
10.1.0.253:/data/diskless/raspbian-lite-base-boot   /boot   nfs   vers=3,nolock   0   2

Network Booting

Now you can hopefully boot. Unless you into this bug, as I did. Where the RPi will sometimes fail to boot. Turns out the fix, which is mentioned on the bug report, is to put bootcode.bin (and only bootcode.bin) onto an SD card. That'll then load the fixed bootcode, and which will then boot reliably.

nz2015 mini-DebConf

Already attending linux.conf.au? Come a couple of days earlier and attend the mini-DebConf too! There will be a day of talks with a strong focus on the Debian project and a bug squashing day.

Debian Miniconf

After 5 years, the Debian Miniconf is back! Run as part of linux.conf.au 2015, this event will attract speakers talking on topics that suit the broader audience attending LCA. The Debian Miniconf has been one of the largest miniconfs in the history of linux.conf.au.

For more information about both these events which I'm organising, head over to: nz2015.mini.debconf.org!

Phew, I've submitted a proposal to run a Debian Miniconf at linux.conf.au 2015 here's hoping that it is accepted!

The Debian Miniconf was held in 2008 in Melbourne, so I feel it is well overdue to run it again.

Well, as of debian-maintainers version 1.39 (uploaded on 2008-07-08) I'm now a Debian Maintainer.

w00t!

Francois will upload a new version of MythTV Status soonish which amongst other things adds the magic fu required to allow me to upload it directly. Oh, and I've changed MythTV Status to be GPLv3.

More w00t!

In our house we have a server that does a number of tasks, one of which is playing a jukebox (using MusicPD) it is hooked up to an amp that has speakers in our study and in our family room. This rocks, turn on the amp, have continuous music that we want to listen to and no chatter.

Unfortunately this doesn't extend to the lounge. Which means that if you sit in the lounge you can only just hear the music (unless you turn it up too loud). I've thought off and on (mostly off) about extending the jukebox to the lounge, the other week Susanne asked about having this, which meant it was finally time for me to really look into possible solutions.

Warning, geeky bits follow...

Since we used to use Icecast2 to stream the jukebox to Susanne's computer in our old flat I tried that first. We have a PC in the lounge that is a MythTV frontend that we could play the stream on, so that aspect is covered. Unfortunately the 5 second lag which was acceptable when Susanne's computer was a long way away from the main jukebox in our old flat was too annoying when the speakers were only a room apart (yes, I could have reduced the buffers, but it would never have been quite right).

So fall back to plan B, stick another soundcard in the jukebox computer and run line level audio to the amp in the lounge. We managed to dig up an old soundcard in the good junk pile (every geek should have a good junk pile), stuck it in the computer and then proceeded to try and configure ALSA to drive it. The plan was to duplicate the same audio stream out to both soundcards. After a bit of researching and a bit of tinkering with asound.conf and it not working I decided not to waste any more time on that path.

I vaguely recalled that PulseAudio was incredibly flexible and allowed all sorts of things. A quick apt-get install pulseaudio pulseaudio-module-hal and bit of reading, and I found that adding the following line to /etc/pulse/default.pa:

load-module module-combine master=<sound card 1> slaves=<sound card 2> set-default-sink combined

Would create a new virtual soundcard output (PulseAudio refers to this as a sink) called "combined" that would replicate the audio to both soundcards. Awesome. (The master and slaves settings were determed by using paman to inspect the PulseAudio devices.)

Then after recompiling MusicPD (to add PulseAudio support, the version in Debian unstable already has it, but stable doesn't), telling it to use that sink and voila, both soundcards are playing the jukebox.

To actually get the line level audio to the lounge I've used some Cat5 we already have running there (with suitable audio flyleads) and we have the jukebox playing in the lounge, in sync with the family room. The sound quality is pretty good, so I won't worry about balanced audio or anything else fancy like that. Gotta love structured cabling.

Another benefit of PulseAudio is we will be able to configure our Linux workstations to use the jukebox soundcards as audio sinks as well, which will mean no more tinny little monitor speakers. w00t!

Update: Say which file to add the load-module line to, and fix a typo.

w00t!

Debian Etch is released!

Now it is time to perform the ritual upgrade dance on a bunch of machines that weren't already running Etch while it was the testing distribution. And also I'll need to decide if the boxes that were already running it should move to lenny (which will is the name for the next stable release). I think I'll defer that decision a few months though...