gemini://jacksonchen666.com/posts/2024-05-01/20-56-54/index.gmi

I run Fedora Linux on laptop-server, and recently upgraded it from Fedora 39 to Fedora 40. Of course, breakages happened, but I did fix them. Although I had one interesting breakage that comes from using `systemctl edit --full`.

I sometimes used `systemctl edit --full` to inspect the full unit file for a systemd unit, and maybe edit them. That's about it. Absolutely nothing interesting to see here. *Foreshadowing.*

In Fedora 40, there is PostgreSQL 16. And in PostgreSQL 16, there is this changelog (found in the PostgreSQL 16.0 changelog):

> Remove symbolic links for the postmaster binary (Peter Eisentraut)

This should be easily dealt with by Fedora Linux (or whoever is maintaining the PostgreSQL package), as they also include the systemd services which starts PostgreSQL. So Fedora Linux (or the package maintainer) should have dealt with this change, right?

The answer is yes. Fedora Linux dealt with that change. So PostgreSQL should work after doing all the upgrades, right?

When I upgraded to Fedora 40, I was met with a need to upgrade PostgreSQL from 15 to 16 (with the pg_upgrade stuff). I did the upgrade as documented in the stdout of the postgresql service. Then I started PostgreSQL. PostgreSQL didn't work. Exit code 203.

That's weird, so I started investigating this whole thing.

After looking around a bunch of stuff later, I found that the postgresql.service unit was trying to execute a binary called postmaster. Which doesn't exist anymore of course, because PostgreSQL has been upgraded to version 16, where the postmaster symlink has been removed.

So why was the new service still calling the old binary?

That's a trick question, as I've deliberately omitted some details about what happened (including the fact that I didn't read the PostgreSQL 16 changelogs at all actually). So here's the details:

The postgresql.service unit file, located in the /etc/systemd/system/ directory, was trying to execute the postmaster binary, which doesn't exist anymore. Also, I don't ever recall creating that file in that directory manually ever. There is another postgresql.service unit file, located in the /usr/lib/systemd/system/ directory, but it wasn't being used because services in /etc/systemd/system/ took priority.

So I removed /etc/systemd/system/postgresql.service after finding that it pretty much matched the file at /usr/lib/systemd/system/postgresql.service (except for the fact that the /etc/ one used postmaster, which isn't desirable anymore), and I was able to start PostgreSQL just fine.

So uh, how did /etc/systemd/system/postgresql.service end up existing on my server? Well, turns out, `systemctl edit --full` is to blame.

What `systemctl edit --full` does is write the full unit into /etc/systemd/system/ (which takes priority over stuff in /usr/lib/systemd/system/), even if you have made absolutely no change. So if the unit file under /usr/lib/systemd/system/ changes, you will not get the updated version of the unit file. That can lead to stuff being outdated and even being behind on breaking changes.

systemctl edit --full: Maybe Don't Use It