Homelab Migration: Part 2, the perils of switching distros

This post was no longer loaded from my house

By the time you read this, the HTML and CSS won't have been loaded from an Intel NUC in my house. I've started the arduous process of migrating my containers to a VPS. This week, I've set up a new Contabo VPS (not a sponsor) on AlmaLinux 9 to act as my outside-the-home-lab during the move process and hit on a few interesting roadbumps I wanted to document.

For the past several years, my Intel NUC has been running, of all things, Ubuntu MATE LTS. The NUC slowly transitioned from a computer I sometimes used as a desktop to a full-time server, and in that transition I never fully removed the desktop environment. I use Ubuntu MATE as my main distribution on all of my other machines; it's a highly-customizable distro with the depth of package availability that comes with Ubuntu, as well as a very stable track record for firmware and drivers across a variety of hardware (mostly).

But I've been transitioning back to RHELalikes for most dedicated server workloads - mostly development environments at work - and since Rocky has only-just released their 9 series, and Contabo has AlmaLinux 9 available, I decided to try re-basing my setup on my preferred production distribution family while I'm migrating anyway.

It turns out that trying a new distribution for the first time in the middle of making live DNS updates for migrations is not the best idea - Alma differs from RHEL more than Rocky does in a few unexpected ways; or at least, Contabo's Alma image does. When I install Rocky from the ISO in a KVM VM, or in GCP and Azure for work, it comes out of the box with SELinux disabled and firewalld enabled. Alma flips those - and while I don't condone disabling SELinux, I had already flipped the DNS for my static sites to the new server so that Caddy could obtain new LetsEncrypt certificates when I realized none of my sites were loading, so I had to make the sledgehammer change of flipping it off to clear the file context issues preventing Caddy from loading simple HTML files.

The lack of firewalld is not as much a problem in that I prefer using firewalld to managing iptables rules manually or with any other front-end, but when I spun up my wg-easy container it restart-looped due to a missing kernel module, iptables-nat. It had been so long since I'd had to add kernel modules on boot that I had to resort to StackOverflow to figure out how to do it the systemd way (add a file to /etc/modules-load.d/). With those tweaks in place, I had a couple of workloads successfully migrated from the NUC to the VPS.

I then migrated the next workload, which is my static-ish photo stream. I've had a lot of Ruby issues with the code that generates the static site, and I had been running a weird mish-mash of a tweaked Docker image that I'd built locally and files being mounted into it from a non-standardized location; I was able in the migration to standardize the file location and move back to the upstream image without the orientation issue I had been experiencing. While the container still takes absolutely ages to start - I don't understand why it has to re-process each photo at start when the volume is bind-mounted - at least adding new photos is less of a mess than it was.

The last workload in this batch was this blog itself. I still haven't figured out a way how to avoid a few minutes of downtime, in this case both to sync the database and to have LetsEncrypt set up, but otherwise it was a very simple lift-and-shift. Ghost continues to be pretty easy to work with, as long as you color inside its lines.

Jordan Cooks

Jordan Cooks

Jordan listens to too many podcasts, has too many streaming subscriptions, loves dogs, is the Integration Engineer Team Lead at Bitwarden, and makes a mean vegan baked mac and cheeze.
North Bend, OR