Migration to TrueNAS

My homelab now runs TrueNAS. I have FreeBSD jails for my Gemini server and Mastodon instance, Transmission instance, and Borg backup repositories. Each jail has its own ZFS dataset within the main storage pool.

TrueNAS offers automatic snapshot scheduling and replication. With this, I've replaced three Borg backup tasks that were carried out over SSH and were rather slow. I have a separate jail and ZFS dataset that contains a Borg repository that my Fedora PC backs up to over SSH.

I find it somewhat refreshing doing things the old-fashioned BSD UNIX way without Systemd. In some way, it feels simpler, more flexible, and less cumbersome. Running a script on a schedule consists of adding a single-line crontab entry instead of writing a Systemd service and timer for it and making sure it's allowed by SELinux. There are some security trade-offs with this approach, as not having SELinux doing access control increases the attack surface. Net traffic to the services running in the jail has to pass through a Suricata gateway first, so hopefully this would sufficiently mitigate any threats.

FreeBSD has a Tailscale package in their repository. I have a tailnet setup between a Linode VPS, where my domain points to and which runs Rocky Linux 9, and the FreeBSD jail that runs my Mastodon instance and Gemini server. Suricata is integrated with Firewalld on the VPS, and it mediates traffic that is forwarded to the tailnet node in the FreeBSD jail.

Setting up and migrating my Mastodon instance to FreeBSD was probably the hardest thing to do here. I ran into some compatibility issues with Ruby 3.1 and the version of Rails that Mastodon recommends, so I had to edit the Gemfile by hand to ensure a specific version of the 'psych' gem is installed by bundler. Installing Mastodon on Linux involves using rbenv to manage the Ruby version; on FreeBSD, the recommended way is to use the latest Mastodon version available in the ports tree, which has Ruby 3.1 as a hard dependency. I suppose it's technically possible to install Mastodon from source via Git and use rbenv to manage the Ruby version, but the Mastodon in the FreeBSD ports tree has instructions that setup FreeBSD-appropriate users, groups, filesystem paths, init services, and permissions.

I've never felt quite satisfied with any homelab setup I've done. There was always something or other that didn't sit right with me and would make me want to re-think the whole setup. I definitely want a next-generation filesystem like Btrfs or ZFS to store all the data with compression and snapshot/rollback capabilities. I want to maintain some extent of security-by-isolation, either with containers or virtual machines, but I've found both of those methods cumbersome on Proxmox or even when using Fedora Server as a sort of hypervisor.

I'd prefer to avoid Ubuntu because of practical and philosophical opinions I have about how they do things; namely, insisting on Snapcraft.io packages over regular DEB packages, Canonical's decision to exclude and discourage Flatpaks in Ubuntu and its flavors, basically forcing the flavors into accepting Snapcraft snaps.

Fedora Server is great and I prefer the Fedora/RHEL ecosystem over anything, but I also prefer to maximize service uptime and minimize reboots, as core system libraries and kernels are updated frequently -- not that this is an altogether *bad* thing, but it's just a lot of upkeep and disruption. I still use Fedora as my daily driver OS. Rocky Linux, Almalinux, or CentOS Stream are stable, long-term enterprise server distributions, so that would minimize frequent service disruption; however, none of them support Btrfs for the root filesystem, and their package repositories are heavily geared toward enterprise computing which means some of the tools that make life easier on the command-line aren't available even with EPEL enabled.

FreeBSD's Jails and BSD UNIX approach to system administration with things like init and task scheduling, along with native ZFS support, seems to be working out pretty well for the time being. FreeBSD's package repositories are pretty up-to-date, and there's also the ports tree. TrueNAS's documentation includes a primer for ZFS among other useful resources, and the FreeBSD Handbook is indispensible as well. I still have a lot to learn about how Jails work and specifically how their features differ from LXC containers (aside from the fact that they don't use cgroups that are specific to the Linux kernel). So, until I happen upon something better, TrueNAS will be my setup for now.

END

Last updated: 2023-05-14

Gemlog archive

hyperreal.coffee