2024-02-28 - Closing the spec gap: mandatory <META> and exponential back-off

There's just a little more than one month to go before my scheduled time for switching to the separated and more formally written protocol and markup specifications and transitioning the current wordier and less formal combined specification into a kind of "technical overview". Sometime between now and then, probably roughly at the half way point, I will make a final update to the current official spec, so as to reduce the difference between it and the new specs, such that the actual switch has very little in the way of consequences for implementers. Two of the largest remaining differences between the specs are discussed here, with a request for comments on various things I might do in response to these differences.

The first is that the current specification is relatively vague with regard to that the part of a response header which comes after the status code - called the <META> in that spec (I honestly don't recall why I chose this name). A maximum length of 1024 bytes is specified in section 3.1, but there is no minimum. However, later in section 3.3, the correct behaviour when receiving a response header with status code 20 and an "empty" <META> is prescribed, so implicitly a <META> of length zero is permitted (note, though, that this empty string still needs to be separated from the status code by a single mandatory space, which just makes these headers weird and ugly). The biggest problem with this is that the correct behaviour when a response header with any other status code and an empty <META> is received is ambiguous. In some cases, e.g. the redirect status codes 30 and 31, there seems to be absolutely no sensible course of action in the face of such a response. In contrast, in the new protocol specification, the <META> part of the response header (not called <META> or indeed anything in that spec) is mandatory for status codes beginning with 1, 2 or 3, while remaining optional for those beginning with 4, 5 or 6. I presume this was intended as something like a bare minimum change to the existing spec in order to rule out what were considered clearly problematic response headers, such as the redirect to an empty string.

I certainly agree that making <META> mandatory for 3x status codes makes sense. It's not quite so essential, IMHO, for 1x, as there's at least a clear fallback here, of presenting a generic "The server has requested input" prompt in whatever language the user's client is configured to use. But rather than debating whether or not to "bless" the change for 1x, I'm rather debating whether or not it isn't just a much better idea to go further than the new spec currently does and make <META> absolutely mandatory regardless of status.

Various people in the Gemini community have already pointed out to me that responses with a status code of 20 and an empty <META> are incredibly rare (that's not just a hunch, it's backed up by the experience of people running large crawler/archiver projects), and also that many existing clients fail to handle those rare responses correctly. Embarrassingly, even my own AV-98 choked on these for many years. I can also say with confidence that my original motivation in the very earliest days of Gemini for allowing empty <META>s with a prescribed default MIME type of text/gemini was that I was worried folks in the Gopher community would balk at every little byte of overhead introduced by the presence of response headers, and making that part optional got the most common response header down to just four bytes (I'm sure I didn't intend for the weird trailing space to be mandatory). In hindsight, this is a ridiculous concern anyway, as TLS overhead dwarfs the header overhead. Even in the absence of TLS, when transferring very small text files as in common in Gopherspace, the overhead introduced by using TCP is non-trivial and greater than that which would be added by including "text/plain" with each response. Besides, Gopher can't be too snooty about "zero overhead" when a substantial proportion of the userbase serves everything as item type 1, adding bogus servers, ports and selectors to each line.

All of which is to say that in the one situation where behaviour around optional <META> is very clearly defined, it is serving no useful purpose, is very rarely used, and is introducing bugs in clients. We can make both the spec and the implementations simpler by making <META> mandatory. If anybody has strong principled objections to this or wants to offer empirical data on how frequently response headers with status codes beginning with 4, 5 or 6 do not include a <META> component, please reach out to me in the coming weeks. Otherwise, I will likely update both specifications to reflect this change in advance of the switching of official status.

The second difference is this one: the current spec defines a "SLOW DOWN" status code 44, where <META> is a number of seconds that automated clients should wait before repeating their request (oh, wouldn't you know, here's yet another case where the current implicit allowance of empty <META> anywhere causes problems). The new spec instead simply says that automated clients should implement an exponential back-off scheme, and the semantics of <META> are the same as any other 4x status code, i.e. it is a (currently optional) human-readable error message.

People in the community have pointed out that the current spec's use of <META> to convey a number of seconds to wait (more or less copied from HTTP's " Retry-After" header) violates what is supposed to be a core design principle of Gemini, namely that simple clients can treat all status codes with the same first digit equivalently without incorrect behaviour or missing functionality. In this case, a simple client might present the user with an error message which is just an integer, which is a recipe for confusion. Now, it is a reasonable expectation that status codes of 44 will only be sent in response to aggressive automated clients and a human user of a simple client is very unlikely to actually encounter this scenario. Nevertheless, it is very unsatisfying having the spec contradict what is supposed to be an overarching principle. Furthermore, I can add that when I actually implemented support for automatically sending 44 responses to my Molly Brown server, I discovered that the standard tool for rate limiting in servers is the so-called "leaky bucket" model, and was soon struck by the realisation that "number of seconds you should wait before sending another request" is actually not a very useful piece of information to send to a client when what the server is really trying to enforce is a maximum average request rate with some leeway for bursts.

So, the current specification for status code 44 is certainly not ideal and the new specification's take seems to be an improvement. What I am wondering is whether or not it is worth specifying something more specific than that automated clients "SHOULD use an exponential back off". I'm not sure what that might be. I am interested to hear thoughts on this matter from people who have already written servers which issue 44 status codes, or who have written automated clients which attempt to honour 44 status codes, or who operate Gemini servers which are struggling against large numbers of automated requests. I suspect there are very few such people out there and I am probably overthinking this, but if you do exist, I am listening. In the absence of clearly superior suggestions in the coming weeks, I will likely change the current specification to match the new specification in advance of the switching of official status.