-- Leo's gemini proxy
-- Connecting to auragem.letz.dev:1965...
-- Connected
-- Sending request
-- Meta line: 20 text/gemini; lang=en
I read an article recently which quoted a post that quoted the Gemini spec at the time about extensibility:
> Another interesting piece of the protocol is that it is explicitly designed to be non-extensible. There is no version number in the protocol, and the response layout was carefully constructed to make extending it hard:
> "To minimise the risk of Gemini slowly mutating into something more web-like, it was decided to [include] one and exactly one piece of information in the response header for successful requests. Including two pieces of information with a specified delimiter would provide a very obvious path for later adding a third piece - just use the same delimiter again. There is basically no stable position between one piece of information and arbitrarily many pieces of information, so Gemini sticks hard to the former option, even if it means having to sacrifice some nice and seemingly harmless functionality"
This author highlights a great point about Gemini's solution to anti-extensibility, which is more restrictive in order to prevent future extensibility. What Gemini gets wrong is that this was not completely necessary, and that Gemini wasn't completely successful in restricting extensibility in every case, although most of its headers are almost completely non-extensible. The biggest hamper to extending Gemini seems to be mainly the community and not necessarily the protocol itself, which is definitely a good thing.
Gemini is in fact extensible in four ways:
Adding information to the mimetype using the ";" delimiter
Adding information to the URL path or using URL parameters
Adding information to the query string
Files
If I wanted to extend Gemini, I could easily do it by adding an additional parameter to the mimetype. This could include a starting byte/seconds for streams, date information, charset, language, version number, and many other things. Typically this would describe the type of the content. This is possible because many mimetype parsing libraries will parse all parameters, whether they are custom or not, since parameters use a consistent syntax of `label=value`. Clients that don't support some parameters will just ignore those parameters.
However, one problem: you don't pass mimetypes in requests, only in responses. This is where the other two extensibility options are possible. A client can pass a UserAgent, a session id, a language, and many other things into both query strings and on the path, although query strings have the disadvantage that clients and servers don't concatenate queries together or use query parameters.
If you want to support extra metadata about files, like abstracts for example, then you could easily have cients take the current path and append the ".abstract" extension onto it and make a request for that file. This could be done for scripts too, and many other things, including favicons ;). In fact, the part of the spec that doesn't allow making automatic requests could be completely ignored by developers if they wish, and there's no way to enforce against this other than users just not using their software. Servers can't tell what clients are being used, so they also can't be rejected. Perhaps the best way to prevent this is to introduce rate-limits into gemini servers that make the experience of clients that automatically make requests slow (by forcing them to sleep in between each request).
Many of these problems were in fact brought up in the mailinglist, and many of these problems are also not solveable, unless we disallow mimetype parameters completely and separate out the lang and charset into their own response fields. Gemini chose not to do this because it introduces a delimiter that, according to solderpunk's thinking at the time, could be used to extend Gemini. However, this assumption is actually incorrect, because which delimiter you choose to use and its relationship to the fields actually matters. This happens to be informed by my experience working on the Misfin(C) proposal where we tried to make misfin less extensible:
In fact, now I see that while we improved upon Misfin(B) and Gemini, we actually didn't make it anti-extensible enough!
To prevent extensibility, Gemini makes the assumption that adding delimiters to requests and responses will allow for extensibility. Here's where that assumption actually works, and it is based on what allowed Gopher+ to work:
<URL><delimiter><lang><CRLF>
in Gopher+, that delimiter would be the tab character. Clients that use the standard spec would be able to parse the above by splitting on the delimiter and checking the first two fields. However, servers could introduce another parameter:
<URL><delimiter><lang><delimiter><script_path><CRLF>
Now old clients can tell the end of the request with CRLF, and the first two parameters with the delimiter, and it will ignore that pesky script_path. But new clients can just read that third parameter. This is exactly how Gopher+ works.
It should also be noted that Gemini responses almost allow for extensibility, because it has a space delimiter with a response header that ends in CRLF. However, because the delimiter is a space, you cannot have any spaces in any of the new fields that one extends the requests with; the delimiter(s) you choose restricts what can appear in your fields. In fact this is why Gemini's responses are actually *not* extensible in *almost* every case, because the second field almost always allows a space character, which prevents one from using the delimiter to attach more fields.
For Gemini requests, because there is no delimiter, everything would try to be interpreted as the URL, and so one cannot add extra fields. The omission of the delimiter prevented extensibility as well as extra request information.
Notice, however, that requests are still extensible because of URL parameters and query strings. If we want to make it less extensible, we have to split the URL into different components that we allow and reject the ones we don't (like URL parameters). And yet, this introduces a delimiter for more than two fields that can be used to extend requests! Whichever way we go has extensibility problems.
So what exactly made the above example's extensibility work? It is three things:
delimiters that can be split on
fields can be ignored because one can spit on a consistent delimiter
the client knows where the end of the header is because the field delimiter is different from the CRLF
There are a couple of different ways we can prevent this extensibility, but I will start with what delimiters we choose to use.
Let us say that we want to have fields but disallow extensibility. We must do this by preventing delimiters from being split on and by preventing this split from allowing clients to ignore extra fields. Gemini's request restriction works (aside from the fact that it uses a full URL) because every character is interpreted as part of one field. If someone tries to add another field, that gets interpreted as part of the URL and breaks clients! Breaking clients is good, because it prevents extensibility.
So how do we add an extra field while breaking clients by any further extensions? If we are adding just one field, then we can actually allow the delimiter in that field:
<URL><tab><field1 that can include tabs><CRLF>
So how does this work first with spec-compliant clients? A client splits on the first tab, and then interprets everything after that first split as part of the second field, including any other tab characters.
What if clients and servers want to add another field, like so?
<URL><tab><field1 that can include tabs><tab><field2><CRLF
It turns out that *they can't*. Firstly, this would break spec-compliant clients because they will interpret field2 as part of field1. Secondly, this also breaks new clients because field1 can have tabs (and any other character), so a new client cannot rely on their being no tabs in the first field.
New clients could get around this by assuming that field1 will never have tabs, but even if they make this assumption, old spec-compliant clients will still break by interpreting field2 as part of field1. It turns out the first way to break extensibility is by allowing only one delimiter that is split on once! This is in fact how Gemini prevents extensibility in *response headers*.
But now what if we wanted more than just one extra field in the spec, what if we wanted another one?
<URL><tab><field1><tab><field2 that can include tabs><CRLF>
The above only works if client implementations only split on the tab twice and ignore any tabs after the first two. This, however, becomes a requirement if it expects that field2 must be able to have tabs. It also expects that field1 doesn't have any tabs now. This is again a similar principle to the previous circumstance: because we allow the delimiter in the last field, this will prevent tabs from delimiting anything else in extended server/clients, and if someone tries to extend it, it will break all spec-compliant clients.
So the first principle is simple:
Require that clients must be able to parse the delimiter character in the last field of the header. This requires that clients only split on that delimiter the number of times that there are fields minus one. In the above example, if we have 3 fields (including the URL), we must only split on the tab twice. Golang and other programming languages do in fact have a mechanism for this. In Golang it's called `strings.SplitN()`.
However, what if we don't have field values that can include the delimiter? Mimetypes don't expect tab characters! And neither do paths! Let's talk about the second principle now:
<URL><tab><field1><space><field2><CRLF>
So now field1 cannot include tabs or spaces, and field2 can include both tabs and spaces. If we split only on tabs, then field2 will be interpreted as part of field1. If we split only on spaces, then field1 will be interpreted as part of the URL. If we split on any number of tabs and any number of spaces, then if field2 contains a tab or a space, it will get split up.
In fact, the only way to possibly extend this is to split once on tabs and split any number of times on spaces and have field2 never include either tabs or spaces, or if we split any number of times on tabs and only once on spaces and field2 never includes tabs or spaces. So here's the second principle:
Inconsistency in the delimiter makes extensibility harder.
In fact we have something like this already in Gemini. No gemini fields or extended fields in responses can include a CRLF because the CRLF is split on *once* to distinguish between the header and the body, and because the body can contain CRLF. So Gemini in fact has two different delimiters in responses: spaces and CRLF. Everything before the first CRLF cannot have CRLFs, but everything after it can. Everything before the first space cannot have spaces, but everything after it, up until the first CRLF, can.
Someone could, however, add extra fields to the front of the gemtext body and clients could read that as metadata. Clients that don't support it would just display it as part of the gemtext body. But this is solely due to how the body is interpreted rather than how it is delimited from the header, and this does in fact break if we try to do this with a body that doesn't expect CRLFs, like binary files. Therefore, the interpretation of field values also matters here just like they do with mimetypes! Since the body is interpreted based on the mimetype, one cannot add metadata to all possible bodies. This is the last principle.
So, yes, Gemini is somewhat non-extensible. But its sacrifice on extra fields is not what makes it non-extensible. One can have more than two fields *and* have anti-extensibility by following the principles from above:
Require that clients must be able to parse the delimiter character in the last field of the header. This requires that clients only split on that delimiter the number of times that there are fields minus one. In the above example, if we have 3 fields (including the URL), we must only split on the tab twice. Golang and other programming languages do in fact have a mechanism for this. In Golang it's called `strings.SplitN()`. *This principle is used by Gemini responses.*
Inconsistency in the delimiter makes extensibility harder. *This principle is also used by Gemini responses, which have two delimiters: spaces and CRLF.*
How fields are interpreted must also be non-extensible. *Mimetypes allow for a semicolon delimiter to specify mimetype parameters, and so fails on this principle. However, Gemini response bodies require that they adhere to the main format specified in the mimetype, so it succeeds mostly on this principle, even if gemtext itself can be extended.*
What Gemini gets wrong is not how it prevents extensibility, but the assumptions about why its anti-extensibility works in the first place.
Here are the next articles in this series:
-- Response ended
-- Page fetched on Mon May 6 20:22:46 2024