-- Leo's gemini proxy
-- Connecting to gemi.dev:1965...
-- Connected
-- Sending request
-- Meta line: 20 text/gemini
2023-00-00 | #tag | @Acidus
I wrote about the Guppy protocol yesterday, which is a smolnet protocol that interestingly uses UDP instead of TCP.
As discussed in that post, I think the choice of UDP may not be the best. However I did have some feedback for @dima on ways to improve the spec while keeping its UDP roots. @Dima updated the Guppy specification to v0.2 yesterday.
I sent @dima a few emails with feedback, and this post lightly edits those and expands on them a bit. My feedback is in order of the sections of the v0.2 specification.
>The fragment component in a guppy:// URL has no special meaning, but clients must not remove it when following links.
Historically, fragments have no meaning to the server, and the majority of URL-based protocols don't send URL fragments to the server (See, HTTP, Gemini, Spartan, etc). Some protocols even say fragments “MUST NOT” be sent. It is counter intuitive that Guppy requires fragments MUST NOT be removed from the URL. Also, this potentially creates another mechanism to provide user input besides query strings, which is odd as well.
Consider changing this so URL fragments are not sent in Guppy requests, to be consistent with other protocols.
Is there a limit on URL length? The specification doesn't say there is. I can fit a 65533 length URL into a UDP packet, including the trailing `\r\n`, but presumably Guppy doesn’t want that.
Historically, not defining a URL length limit has been the source of problems in protocols, such as with HTTP. Different web browsers had different undocumented URL limits which would make sites act inconsistently. Generally the rule in HTTP land is to have URLs < 2048.
Consider adding a URL limit to Guppy of 1024 or 2048.
> The client must support at least text/plain and text/gemini ... documents.
What does "must support... text/gemini" mean? Gemtext is just text, with some special lines. Do this mean a client must have clickable/selectable hyperlinks? That requirement would mean that something which just displayed the text like "curl" would not be considered a client.
Consider removing this entirely.
> text/gemini (without the Spartan := type)
I discuss this in more detail below, but given the way error packets are used to tell the client about needing user input, I think Guppy would benefit from Spartan-style `:=` link lines.
> "If encoding is unspecified, the client may assume it's UTF-8 or ASCII."
A client cannot assume a response is 2 different character encodings. UTF-8 and ASCII are different things, though UTF-8 is a superset of ASCII. If a response contains emoji, but a client assumes its encoding is ASCII instead of UTF-8, the response will not render correctly. Also, encoding only makes sense on “text/*” responses so it should only be assumed/inferred on text/* MIME types.
Consider updating this to say "If encoding is unspecified on text/* MIME types, the client must assume the encoding is UTF-8."
The specification does not say how to specify response text encoding. I assume it's via the `charset` parameter on the MIME type.
Considering adding: “Encoding is specified using the `charset` parameter on a MIME Type. The `charset` parameter on non-text MIME types should be ignored."
>In Guppy, all URLs can be (theoretically) accompanied by user-provided input. The client must provide the user with means for sending a request with user-provided input, to any link line.
> Server and content authors should inform users when input is required and describe what kind of input, using the link's user-friendly description.
> If input is expected but not provided by the user, the server must respond with an error packet.
I found the system for the client to inform the user that user input is required confusing. Using an error packet to tell the user they need to request a URL with input mixes telling the user there was an error (e.g. something broke) with an instruction to the user (hey, try this again, but first prompt them for input). How does the client which of these options an error packet means?
I suppose this could be done with the error packet's error message, but:
that is language specific
that makes writing a client harder.
A client has to figure out from the error message that an error packet really is a “hey, you need to give me input" packet, and then presumably the user would have to do something to get an input field to use.
Make input be denoted by an error response also also means that an the author would need to configure the Guppy server to return this error, for every URL that needs user input. This sounds onerous, and its possible the content authors don't have access to the server configuration (think pubnixes).
If Guppy wants content authors to be able to inform users that input is required, I feel a Spartan-Style `:=` line type would be the best solution. Consider these two experiences:
Using an error packet to signify user input:
user downloads gemtext
user clicks link
user gets error message telling them they need to provide input for that link to work
user clicks “back” button
user figures out how client lets them submit that link, but with user input of some kind
user gets response for user input
Using a `:=` line:
user downloads gemtext
user clicks link using a := line
client prompts user for input, and requests URL with that user input
user gets response for user input
You could still have the Guppy server return an error if a URL that requires user input was requested without user input (like a malformed link or something), but augmenting this with a `:=` link line allows the client to know ahead of time that a link requires user input.
> Clients may cache the returned error and present it to the user on attempt to access the same page without providing input.
This is one of the few parts of the spec that mention caching. I talk about caching later, but I don't think you want errors cached in general.
I've actually really come to appreciate Gemini's design with status code 10. It seems wasteful at first, but being a status code allows someone to build a system that accepts input, redirects you to another URL, and prompt for another input. Spartan can't do this because prompting for input happens only when the user clicks on a link. The := is kind of a nice shortcut to tell a client that a URL will have a prompt, but shouldn't be the only method to get user input. Honestly, I would just copy Gemini's use of a Status code to prompt for input.
> Implementers should use encrypted file formats for input and downloaded content, and arbitrary or even randomized paths (for example, /a.gmi rather than /my-post-about-chess.gmi), if eavesdropping is a concern.
Advice about randomized paths is... certainly a creative solution but I'm not sure what it gains you. An eavesdropper could still see the content, and if eavesdropping is a concern, the user shouldn’t be using Guppy at all. Full Stop. I feel the specification should not give suggestions or imply that Guppy can be used in a secure or private way.
> Implementers should use external means, like digitally-signed documents or PGP, as proof of authenticity.
Security recommendations in standards tend to age extremely poorly and I suggest not including this. A good example is PGP, whose web-of-trust model works pretty poorly in practice.
Consider removing the Security and Privacy section entirely, or replace it with:
> "Guppy is an insecure protocol by design. It makes no guarantees about the integrity of any content, the confidentially of any traffic, or the veracity or identity of any servers."
> Clients and servers may restrict packet size, to allow slower but more reliable transfer. However, the packet size limit must be at least 512 bytes.
Why is there a minimum packet size at all? How can I send content < 512 bytes? Do I have to pad it? With what character? How does a client know where content ends and padding begins, since there is no concept of Content-Length?
Also, it's not clear if by “packet size” the specification means the entire UDP packet, or the data in a Guppy packet (UDP packet size minus the size of the status line)?
Consider updating this to say “A Guppy packet must fit inside a UDP packet. Guppy packets with a status line but no data have special meaning, as discussed below”.
> Client and servers should handle timeouts gracefully and must distinguish between timeout and proper termination of the session.
This is very vague. How long should a client wait for an EOF before resending an ACK? How long should a server wait for an ACK before resending? What does "gracefully" mean? How does a client distinguish between timeout and proper termination of the session?
Consider adding details about this here, or in the sections below about packets.
> url\r\n
What is the character encoding used to send the request?
> The query part specifies user-provided input, percent-encoded.
Is only the query string part URL encoded? What about special characters in the path? Is space allowed to be encoded as a `+` or must it be `%20`?
Consider punting on all of this and have Guppy follow the same URL character encoding and URL encoding rules that HTTP uses:
ASCII encoding string
URL encoding applied throughout
Spaces can be encoded as `+` or %20
Domains with Non-ASCII characters are handled as puny code
The Gemini specification's use of “UTF-8 encoding for URLs” has lead to a host of problems:
How to handle international domain names
How URLs should be represented in gemtext documents vs how they are transmitted in a Gemini request.
The use of UTF-8 for URL encoding was a real sticking point when trying to get Gemini support into curl and remains one of the few confusing parts of the Gemini specification.
There is no guarantee that a request packet makes it to the server. While later parts of the specification talk about re-sending packets, nothing is mentioned about when to re-send a request packet. How long should a client wait before resending a request? Should requests be resent? How does a client know the different between a request that needs to be resent and a server that just doesn't exist at that port or is having problems?
Resending a request also has the potential to be much more problematic that resending an ACK or a continuation packet. A request, especially with user input, could be making changes to the server (submitting a post, appending to an address book, etc). I'm not sure the best way to handle this, short of having 2 different protocols: one that implements a primitive reliable transport layer on top of UDP, and a request/response application layer that runs on top of it. See details below.
> The sequence number is an arbitrary number between 2 and 2147483647
Starting at 2 is presumably to make it easier for a client to distinguish a sequence number from a error or redirect response, but it really doesn’t help. A sequence number of 1014 would look like an error (it starts with 1) but isn’t.
> minus the number of packets needed to transmit the response.
A server often don’t know how many packets something with take when its start sending data, so the "minus the number of packets" part is not helpful.
Consider just saying “A sequence number is a positive number between 1 and 2^31” and leave it at that. The specification talks about sequence numbers increasing later anyway.
Personally, to implement a server, I would code the server would pick a random number between 1 and 2^16 on reach incoming request as the initial sequence number, and increment it with each packet. That way the server won't run out of sequence numbers when transmitting any reasonably sized response.
> Success Packet Format:
> seq type\r\n$data
Consider using a status code (like 2) for Success, just like Guppy does for errors and redirects, so the format would be:
2\s[sequence number]\s[mimetype]\r\n[$data]
This makes it way easier when implementing a client. The first response packet to a request will then always be for the form `[0-9]\s`.
Right now, a Success packet cannot be empty. It has to the MIME type and some data. Having an empty success packet (e.g. just a MIME type) would make it easier to write a server. Under the current design, a server needs to have all the bytes it needs to include in a UDP packet before the can send the packet. This means that for the server to reply to a request with a Success Packet, it would have to receive the request, find the file on disk (at least some of it), grab the first so many bytes, combine those with the bytes the representing sequence number and mimetype, and then send that packet to the client.
If empty success packets are allowed, a server could immediate respond to the client that it can service a request, and then do the work to go load the file, or generate the appropriate response.
> The client must acknowledge every success, continuation or EOF packet by echoing its sequence number back to the server.
This makes Guppy an unreliable protocol. Redirects and Errors are not ACKed by the client, so the server has no way to know if the Redirect or Error actually made it to the client. Acknowledgements would need to be sent by the client for all packet types.
Also, the specification says that an Acknowledgement packet contains just the sequence number, but in the "Examples" section, all the examples show the acknowledgement packet for a Success packet containing both the sequence number and the MIME type that it is ACKing. I think this is mistakingly carried over from the Guppy v0.1 spec.
> The server should wait for the client to acknowledge the previous chunk of the response (the success packet or the previous continuation packet) before sending the next continuation packet, to avoid waste of network bandwidth.
Basically, Guppy only allows one data packet to be in-flight at any time. The server sends a packet that requires an ACK. The server waits until it gets an ACK. If the server doesn't get an ACK, the server resends the packet. The client sends an ACK for certain packet types. If the client doesn't receive what it is expecting, it resends an ACK.
> The server may attempt re-transmission of a success, continuation or EOF packet after a while, if not acknowledgement by the client.
> The client may attempt re-transmission of an acknowledgement packet and the server must ignore acknowledgement packets it's not waiting for.
Consider adding some guidance here. How long should a server wait before re-transmitting? How long should the client wait before resending an ACK? See also the question about about what to do with a Request packet that doesn't make it to the server.
> 0 url\r\n
What is the character encoding of the URL? See comments about URL character encoding above.
> The URL may be relative.
This is vague. Can the URL be a fully qualified URL? A protocol-relative URL? (e.g. //example.com/foo).
Consider replacing this with "URL is an ASCII encoded URL, and can be any valid URL (e.g. fully qualified, relative, etc.)"
The specification doesn't mention if a redirect can redirect a client to another host or another protocol. Gemini allows cross-host redirects without a prompt, and cross-protocol redirects with a user prompt. Spartan doesn't allow cross-host or cross-protocol redirects at all.
Consider being more specific here.
> The client may remember redirected URLs. The server must not assume clients don't do this.
This appears to be mean caching. If so, then say caching. I think if Guppy allows caching redirects, it prevents some cool options. For example, someone creating a CGI that redirects visitors to random Guppy pages each time it is requested won't work with redirects can be cached.
> 1 error\r\n
What is the character encoding of the error? The error message is displayed to the user, so conceivably it will need to support being in different languages.
Here are additional thoughts I had while building a toy Guppy implementation and reading v0.1 and v0.2 of the Guppy specification.
Guppy's UDP nature means that implementations are more complex, because they need to handle retransmissions, missing packets, etc. However testing that an implementation is properly handling these cases is super challenging. When writing a client and server on localhost, it is not easy to "drop this ACK and see that the server responds properly". I'm sure I could do something with ifconfig, or a firewall, but those are coarse-grain tools that drop or block entire classes of traffic. There are traffic shaping kernel modules that let you add a % of packet loss to a link, but that's random packet loss. It is extremely difficult to test a client or server properly handles a specific missing packet scenario.
I really had no easy way to test my implementation, other then putting the server on the internet and just "trying it." I have no confidence that either my client or server can successfully handle all failure modes.
Right now 0 and 1 are used as status codes, which really isn't needed (see comment about sequence numbers and the Success packet). Guppy should consider using Status Code values that better align with existing protocols. For example, Spartan uses `2`, Gemini uses `20`, and HTTP uses `200` for success, so consider using `2` for Guppy's success. I suggest `3` for redirect and `5` for an error.
The specification mentions caching in a few places, but doesn't provide guidance on length of time to cache, or what should be cached and what shouldn't be cached. Here is what I suggest:
Redirect response - Cannot be cached
Error response - Cannot be cached
Success response - Could be cached for a short time window, purely to make a client feel better. Short means minutes, not hours or days. Clients that do cache success responses MUST provide a "hard reload"-type function that forces the client to re-request the page.
Is "page" or "response" even the right word? Maybe document? We don't want to cache individual packets here. I'm trying to talk about caching the entire response/document, not the individual pieces. Speaking of confusion with terms...
Guppy's terminology is a confusing and not well defined. I'm sending UDP packets, but Guppy also defines "packets". Guppy doesn't really define what a completed document should be called, or a request and a response. These exist "above" the packets that are used to assemble it. The "status line" terminology is confusing as well, since outside of Success, Redirect, or Error packets, it doesn't represent a "status" at all.
I struggled with what to name variables and functions when writing my code, because it wasn't clear what different parts or concepts were called in Guppy.
While writing a Guppy client and server, I found the mixing of transport layer commands (ACK, Continuation, EOF) and application layer commands (Success, redirect, error) confusing. While writing code, I had to keep thinking “Is this packet something that gets surfaced out of the reader layer, or is this just something internal to mean that the reader needs to keep going?"
The way I have seen applications implement UDP-based protocols in the past is they actually write 2 protocols:
One protocol that uses basic re-transmission, acknowledgement, and closing logic on top of UDP to implement a simple reliable transport layer
A different application protocol that runs on top of that reliable transport layer.
Of course, having 2 protocols adds complexity to the spec.
As mentioned above, parts of Guppy v0.2 are unreliable. Client don't know if a request was received or not, and whether to resend it. Errors and Redirects don’t get ACKs, so servers have no way to know if a client got them. Guppy can and should add more ACK requirements.
I want to call out resending request packets specifically, because there are huge implications if the request contains a query string since that could be making changes on the server!
However, assuming Guppy fixes all that, what is the end goal? It appears that Guppy is just slowly re-implementing features of TCP, but in a less powerful or clear way. And each iteration of this just makes the protocol harder to implement.
For example, Guppy has sequence numbers, but does not go the full way to allow for out-of-order retrieval and ACKing. Since only 1 data packet should be in flight at a time, the only really purpose sequence numbers is to to guard against duplicate or retransmitted packets that are arriving very late, after the client and server have both moved on on the next data packet that is in flight.
I really do appreciate @dima's work. It is hard to design protocols, and they are doing it in public as well, which is really awesome.
In its current (v0.2) form, Guppy is a nice fun personal project or a learning exercise in protocol design, but I don't see a practical future for the follow reasons:
Guppy is more complex than other existing unencrypted protocols (Spartan, gopher, nex, finger)
Gubby v0.2 and below are unreliable, as major parts like requests, redirects, and errors don't require acknowledgement.
It is difficult to test the accuracy or robustness of a Guppy implementation due to the challenge of simulating the behavior of UDP (dropped packets, delayed packets, duplicate packets, etc).
That said, I would strongly encourage people to read the specification and play with it. It could be that, much like writing bubble sort, the value of Guppy is what you get from the experience of writing a Guppy implementation. In which case, none of my feedback above matters. 😂
-- Response ended
-- Page fetched on Tue May 21 18:38:17 2024