Switching to Vger on FreeBSD

Author: Ben <benk@tilde.team>

Sat Jul 23 07:40:22 AM +05 2022

You might have noticed yesterday that my capsule was down for some time. I'm sure most people didn't notice because it wasn't down for that long, but while looking at the access log to my Gemini server, it seems to get requests near constantly. (Although, possibly the majority of requests are for my twtxt.txt file, which may be using more resources than it's worth!)

For a long time I wanted to try vger on my server after reading about its design. Its simplicity and putting security first made it sound very attractive to me. With geminid, security had sometimes been a concern, although I was confident that my setup was safe enough. For example, I had been running geminid as its own user with limited permissions, and the author had also long ago added code to stop geminid from straying outside its content directory, but an advanced hack might still have been able to circumvent that for all I know.

Vger, on the other hand, seemed much more secure. That's not to say geminid is insecure, but perhaps it's simply normal. Also it hadn't been updated in a long time (likely, it doesn't need to be), so I wouldn't call geminid unmaintained, but I suppose its development has reached a conclusion. Also, I found it attractive that vger utilized security features native to BSD, though I don't know those are being used on my FreeBSD setup.

After noticing that vger is one of only a few Gemini servers available prepackaged for FreeBSD, I really wanted to try it. However, after several failed attemtps I all but gave up.

One major problem, was that vger needs an external program to handle TLS connections. It was oiriginally made to use relayd for this, which apparently is specific to OpenBSD. FreeBSD has some version of relayd available, but a very old one, and either the outdated version couldn't do what vger needed, or it could but how to configure it was a mystery since the configuration used by vger wasn't working with that version, it seems. Maybe the port has other issues too, but for me that was a show-stopper.

Thankfully, nginx, which I already had on my server, is able to perform this function, and apparently it does so very well. Solene provided an example configuration for nginx, which turned out to run perfectly!

The next issue was getting vger to run at all using FreeBSD's inetd. What I found out the hard way is that FreeBSD's inetd doesn't let you define services on arbitrary ports. Services must be defined in /etc/services, and that wasn't too hard to do, but I just had no idea that needed to be done. After doing that and fixing inetd.conf, vger was then serving my capsule.

There was a final problem, however, and this one is a bit more complicated and severe. After getting vger to run, my login shells were constantly getting spammed by alerts from FreeBSD's syslogd whenever vger was receiving a request. Since my server gets so many requests, it made my sessions unusable.

After hours of frustration trying to deal with syslogd, I finally discovered that this behavior is somehow unique to vger itself. I thought it was inetd doing this, but apparently it doesn't generate such messages. I also could not figure out how to get syslogd to suppress them. Instead--and this seems like a hack--I was able to run vger with the -u option, which changes its behavior significantly enough that for whatever reason syslogd no longer gets messages from vger. It doesn't make sense, but it works.

This seems like a problem that is not easily solved, and is either the fault of syslogd, my fault for not knowing how to work syslogd, vger's own fault, or something the FreeBSD port maintainer should fix. None of these are likely to change any time soon, so at the moment it looks like I'm stuck with the configuration I got!

For now, it seems to be working very nicely. I hope that this increases the uptime of my server significantly, because now the service runs automatically when FreeBSD boots. In the past I had to load geminid manually because I never bothered to create an init script for it. (Of course, I don't know how to do that, though it shouldn't have been too difficult.) I usually never forgot to do this, but sometimes for example if the server rebooted due to a power failure, I would not even notice it had rebooted. In the worst case, my capsule was down for days without me noticing!

Of course, I'm glad my server is able to turn itself back on automatically when power returns, but I obviously didn't have my Gemini server set up to take advantage of this beneficial behavior and load automatically. Now that this is resolved, the only thing that can stop my server from serving Gemini is if the power goes out or my ISP goes down.

Aside from the syslogd issue, another resulting problem from my setup now is that I no longer have logs of which files are being requested on my capsule. I can now more accurately measure how much data Gemini is transferring in total, but not the data used by specific things like my gemcast. Even then, with geminid I could only estimate this as well. With my current logging, I can do some similar estimation like consider transfers above a certain size to be belonging to the gemcast. Bytes transferred could uniquely identify some files as well, given that the transfer completes. (For small files, it usually should.) This should be enough.

So far, it seems my gemcast is using a lot of data! July was another record month after May as being the busiest on record. In May, gemcast recordings were requested 348 times to a total of 8.2 GB, and in July (which isn't over yet!) it has stood at 8.6 GB for 399 files.

Will I continue using vger? Probably yes. FreeBSD also provides gemserv and gmid. I would rather stick with vger and hope any outstanding issues will be resolved over time. Aside from the logging fiasco, it appears to be serving my capsule perfectly. Most importantly, audio streaming still works, so we are good to go!

Perhaps eventually I will also begin experimenting with CGI, which vger supports, but I think geminid did not. (Not that I needed this, but it could be fun, right?)