Andrew's blog (Entries tagged as linux)

Let's Encrypt with Octavia in OpenStack

andrew@etc.gen.nz (Andrew Ruthven) — Sun, 23 Oct 2022 05:09:00 +0000

I like using Catalyst Cloud to host some of my personal sites. In the past I used to use CAcert for my TLS certificates, but more recently I've been using Let's Encrypt for my TLS certificates as they're trusted in all browsers. Currently the LoadBalancer as a Service (LBaaS) in Catalyst Cloud doesn't have built in support for Let's Encrypt. I could use an apache2/nginx proxy and handle the TLS termination there and have that manage the Let's Encrypt lifecycle, but really, I'd rather use LBaaS.

So I thought I'd set about working out how to get Dehydrated (the Let's Encrypt client I've been using) to drive LBaaS (known as Octavia). I figured this would be of interest to other people using Octavia with OpenStack in general, not just Catalyst Cloud.

There's a few things you need to do. These instructions are specific to Debian:

Install and configure Dehydrated to create the certificates for the domain(s) you want.
- apt install barbican
Create the LoadBalancer (use the API, ClickOps, whatever), just forward port 80 for now (see sample Apache configs below).
Save the sample hook.sh below to /etc/dehydrated/hook.sh, you'll probably need to customise it, mine is a bit more complicated!
Insert the UUID of your LoadBalancer in hook.sh where LB_LISTENER is set.
Create /etc/dehydrated/catalystcloud/password as described in hook.sh
Save OpenRC file from the Catalyst Cloud dashboard as /etc/dehydrated/catalystcloud/openrc.sh
Install jq, openssl and the openstack tools, on Debian this is:
- apt install jq openssl python3-openstackclient python3-barbicanclient python3-octaviaclient
Add TLS termination to your LoadBalancer
You should be able to rename the latest certs /var/lib/dehydrated/certs/$DOMAIN and then run dehydrated -c to have it reissue and then deploy a cert.

As we're using HTTP-01 Challenge Type here, you need to have the LoadBalancer forwarding port 80 to your website to allow for the challenge response. It is good practice to have a redirect to HTTPS, here's an example virtual host for Apache:

<VirtualHost *:80>
    ServerName www.example.com
    ServerAlias example.com

    RewriteEngine On
    RewriteRule ^/.well-known/ - [L]
    RewriteRule ^/(.*)$ https://www.example.com/$1 [R=301,L]

    <Location />
        Require all granted
    </Location>
</VirtualHost>

You all also need this in /etc/apache2/conf-enabled/letsencrypt.conf:

Alias /.well-known/acme-challenge /var/lib/dehydrated/acme-challenges

<Directory /var/lib/dehydrated/acme-challenges>
        Options None
        AllowOverride None

        # Apache 2.x
        <IfModule !mod_authz_core.c>
                Order allow,deny
                Allow from all
        </IfModule>

        # Apache 2.4
        <IfModule mod_authz_core.c>
                Require all granted
        </IfModule>
</Directory>

And that should be all that you need to do. Now, when Dehydrated updates your certificate, it should update your LoadBalancer as well!

Sample hook.sh:

deploy_cert() {
    local DOMAIN="${1}" KEYFILE="${2}" CERTFILE="${3}" FULLCHAINFILE="${4}" \
          CHAINFILE="${5}" TIMESTAMP="${6}"
    shift 6

    # File contents should be:
    #   export OS_PASSWORD='your password in here'
    . /etc/dehydrated/catalystcloud/password

    # OpenRC file from the Catalyst Cloud dashboard
    . /etc/dehydrated/catalystcloud/openrc.sh --no-token

    # UUID of the LoadBalancer to be managed
    LB_LISTENER='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'

    # Barbican uses P12 files, we need to make one.
    P12=$(readlink -f $KEYFILE \
        | sed -E 's/privkey-([0-9]+)\.pem/barbican-\1.p12/')
    openssl pkcs12 -export -inkey $KEYFILE -in $CERTFILE -certfile \
        $FULLCHAINFILE -passout pass: -out $P12

    # Keep track of existing certs for this domain (hopefully no more than 100)
    EXISTING_URIS=$(openstack secret list --limit 100 \
        -c Name -c 'Secret href' -f json \
        | jq -r ".[]|select(.Name | startswith(\"$DOMAIN\"))|.\"Secret href\"")

    # Upload the new cert
    NOW=$(date +"%s")
    openstack secret store --name $DOMAIN-$TIMESTAMP-$NOW -e base64 \
        -t "application/octet-stream" --payload="$(base64 < $P12)"

    NEW_URI=$(openstack secret list --name $DOMAIN-$TIMESTAMP-$NOW \
        -c 'Secret href' -f value) \
        || unset NEW_URI

    # Change LoadBalancer to use new cert - if the old one was the default,
    # change the default. If the old one was in the SNI list, update the
    # SNI list.
    if [ -n "$EXISTING_URIS" ]; then
        DEFAULT_CONTAINER=$(openstack loadbalancer listener show $LB_LISTENER \
            -c default_tls_container_ref -f value)

        for URI in $EXISTING_URIS; do
            if [ "x$URI" = "x$DEFAULT_CONTAINER" ]; then
                openstack loadbalancer listener set $LB_LISTENER \
                    --default-tls-container-ref $NEW_URI
            fi
        done

        SNI_CONTAINERS=$(openstack loadbalancer listener show $LB_LISTENER \
            -c sni_container_refs -f value | sed "s/'//g" | sed 's/^\[//' \
            | sed 's/\]$//' | sed "s/,//g")

        for URI in $EXISTING_URIS; do
            if echo $SNI_CONTAINERS | grep -q $URI; then
                SNI_CONTAINERS=$(echo $SNI_CONTAINERS | sed "s,$URI,$NEW_URI,")
                openstack loadbalancer listener set $LB_LISTENER \
                    --sni-container-refs $SNI_CONTAINERS
            fi
        done

        # Remove old certs
        for URI in $EXISTING_URIS; do
            openstack secret delete $URI
        done
    fi
}

HANDLER="$1"; shift
#if [[ "${HANDLER}" =~ ^(deploy_challenge|clean_challenge|sync_cert|deploy_cert|deploy_ocsp|unchanged_cert|invalid_challenge|request_failure|generate_csr|startup_hook|exit_hook)$ ]]; then
if [[ "${HANDLER}" =~ ^(deploy_cert)$ ]]; then
    "$HANDLER" "$@"
fi

Laptops and networks

andrew@etc.gen.nz (Andrew Ruthven) — Tue, 28 Jan 2014 09:32:36 +0000

Back in the old days, we had workstations. And only workstations. They lived on a network, and having them work in that network was simple. Printers just worked (thank you printcap), network shares just worked (thank you NFS) and life was good.

Then along came laptops. We wanted to be more mobile, using our laptops on different networks or even without a network! No one wanted hardcoded printers anymore, or network shares defined in /etc/fstab. Using an Automounter was an option, but if you were on a different network then having the Automounter around would stall tools like nautilus and file indexers etc.

So we need something which can start up relevant services when you connect to a network, and then stop them when you leave that network.

To support this, a few years ago I wrote a NetworkManager dispatcher.d script to do just that. When you connect to a specific network (using the NetworkManager UUID or a specific gateway MAC) or a VPN connection then autofs is started, users GTK bookmarks have any bookmarks for their Network shares added and CUPS is restarted.

When the connection goes away, then autofs is stopped, any GTK bookmarks for the Network shares are removed and any mounts for the Network shares are lazily unmounted.

I'm not sure if this will of use to anyone else, but if it is I'd love to hear from you. You can browse the code or clone the repo.

Included are sample autofs config files, the dispatcher, and the tools for managing the GTK bookmark files.

LCA2010 - Call for Papers - closing soon!

andrew@etc.gen.nz (Andrew Ruthven) — Tue, 21 Jul 2009 21:21:42 +0000

Wow, the linux.conf.au 2010 team have had a an awesome response to our Call for Miniconfs with over 32 amazing submissions. But the work isn't over yet!

Our Call for Papers is still open, but closes on Friday. That's right, Friday. If you'd like the opportunity to speak at one of the best FOSS conferences in the world, this is your chance, we want to hear from you!

For more information about submitting a paper, have a look at the Papers Info page on our website.

MythTV Status in your MOTD (and email) - Version 0.6.1

andrew@etc.gen.nz (Andrew Ruthven) — Thu, 13 Dec 2007 19:19:53 +0000

I'm pleased to announce a new version of my little script - 0.6.1.

Notable new features:

Details on disk space are reported, and individual storage groups[0];
If there isn't enough program guide data, a warning is shown;
The --email-only-on-conflict option has been extended to mean "Only on alert" which now includes both the previous points (this means, you'll only be emailled if there is something which needs your attention);
For each recording (scheduled, current and conflict) show the channel name it is related to;
We do a little bit of rinsing on the XML data now to remove some invalid unicode characters that occasionally appear;
Suppress some warnings if the user doesn't have access to MySQL and handle some other error conditions in a nicer manner.

If you install the .deb, then the MOTD is updated automatically and the email will be sent daily if there are conflicts (and you've set some email addresses).

As usual, you can check the project page.

[0] To get the details on the individual storage groups you need to apply a couple of patches to MythTV. Please see the FAQ in the tarball or for Debian type boxes, /usr/share/doc/mythtv-status/FAQ .

Update: I've disabled comments for now as I'm tired of all the spam.

MythTV Status in your MOTD (and email) - Version 0.5.1

andrew@etc.gen.nz (Andrew Ruthven) — Thu, 22 Nov 2007 19:28:12 +0000

I'm pleased to announce a new version of my little script - ~~0.5~~ 0.5.1.

Notable new features (from the shiny new ChangeLog):

Add support for printing colour in the encoder status display.
Show any schedule conflicts, or if the MythTV Perl API isn't usable, a warning is displayed.
Display the amount of time until the next recording.
Add --version & --help options.
Allow suppressing blocks of output.
The report can be sent via email with the subject set correctly. With an option to only send out email if schedule conflicts are detected.

If you install the .deb, then the MOTD is updated automatically and the email will be sent daily if there are conflicts (and you've set some email addresses).

As usual, you can check the project page.

Update: I've released 0.5.1 which should make the colour look good on non-black backgrounds as well.

MythTV Status in your MOTD

andrew@etc.gen.nz (Andrew Ruthven) — Thu, 15 Nov 2007 20:24:31 +0000

The other day I was recompiling some kernels during the day on our MythTV backend, which at the time happened to be recording my wifes favourite program (Grey's Anatomy, we discovered it part way through the season, so the repeats are good). It turns out the compilation caused some issues with the recording (which reduced the WAF).

So to increase the WAF, I'm making sure this doesn't happen again. I've written a script that spits out the MythTV status which is then displayed in the MOTD. It is available over here.

I've found this is a pretty effective way of letting me know of any upcoming recordings and what is going to be happening on the system in the next little while. It is a whole lot easier than using the status page on MythWeb.

Sample output:

MythTV status for localhost
===========================
Status: Mon Oct 15 2007, 8:41 AM

Encoders:
cerberus (9) - Idle
cerberus (12) - Idle

Scheduled Recordings:
2007-10-15 13:30:00 - Grey's Anatomy
2007-10-15 18:00:00 - 3 News
2007-10-15 20:30:00 - Mythbusters
2007-10-15 22:00:00 - Flight Of The Conchords
2007-10-16 13:30:00 - Grey's Anatomy
2007-10-16 18:00:00 - 3 News

I run Debian, so there is a .deb. For those not running Debian there is a tarball (but the MOTD updating glue is Debian specific at the moment). For those interested in hacking there is a git repository.

Hopefully I'm not the only one that finds this useful.

IPv6 Firewalling

andrew@etc.gen.nz (Andrew Ruthven) — Wed, 14 Mar 2007 22:02:10 +0000

Firewalling IPv6 on Linux seems to be a vaguely documented topic, and most of that documentation seems to be out of date as it is a fast moving target. I've spent a bit of time over the last couple of days working on improving my firewalling situation and thought I should write up what I've found.

After a bit of digging I found that while IPv6 connection tracking was merged in 2.6.16, the configuration options are somewhat hidden. Up until yesterday I was running 2.6.19.x on my firewall and I discovered that while ip6tables allowed me to configure a stateful firewall, it wasn't actually doing anything!

I looked around for the required nf_conntrack_ipv6 module and couldn't find it. I looked in my running kernels config and couldn't find it. In fact I couldn't find any option for enabling IPv6 connection tracking at all. After some digging (grep'ing the Kconfig files helps) I found that I needed to change over to the new (experimental) Layer 3 Independent Connection tracking support.

The catch here is that if you have the old school Connection tracking (CONFIG_IP_NF_CONNTRACK) enabled you'll never see the new independent method (CONFIG_NF_CONNTRACK) in menuconfig. Which is why I'd never seen it before. So I disabled CONFIG_IP_NF_CONNTRACK (in IP: Netfilter Configuration), enabled (the now visible) CONFIG_NF_CONNTRACK (in Core Netfilter Configuration) went into both the IP and IPv6 Netfilter Configuration menus and selected support for the connection tracking option.

Compiled, installed and rebooted. Suddenly I had IPv6 connection tracking working. w00t! But no IPv4 NAT. Damn. It turns out that IPv4 NAT support was only ported to the new Layer 3 Independent Connection stuff in 2.6.20.

So I downloaded 2.6.20.3, jumped into the IP: Netfilter Configuration menu and found "Full NAT". That's what I want. Compiled, installed and rebooted.

Now I have my old IPv4 NAT working, and a full stateful IPv6 firewall (with no NAT!).

Oh, if you are using IPv6 stateful firewalling with Linux then you want to upgrade to 2.6.20.3, it fixes an issue with incorrectly classifying IPv6 fragments as ESTABLISHED and letting them through. Oops. Also, 2.6.20 moves the config options around again...