Retrospective on developing diohsc

Diohsc is feature-complete, my TODO list is literally empty, and I think I'm about ready to finally make a first release. I'll leave it to settle for a week first, but I thought now would be a good time to write up a little retrospective on how it came to be.

Creation story

This was not how I meant to spend my free time in these months.

I started diohsc after reading the gemini protocol, seeing that implementing a client was meant to be a "weekend project", noting that I had an hour of free time, and deciding to treat this as a challenge. Could I hack a functional client together in an hour?

The answer turned out to be... "no". I spent most of that hour reading the documentation on the haskell tls module. But by the end of the weekend I did have a minimally functional client, in 162 lines of haskell.

Then... I kept working on it. I find haskell programming at least as addictive as the worst=best computer games. The compiler and type system have your back, so there's none of the fear of making a typo which could lead to hours of bughunting which you get with C, and the way the types slot together so neatly makes finding the right concise expression for something very much like solving a satisfying puzzle. So having started, I didn't want to stop.

And there was plenty to do, despite the simplicity of the protocol and text format, and even though I made avoiding feature creep a priority from the start. First there was TOFU to think about, and later client certificates, and still later TLS sessions and 0RTT. These involved wading my way into TLS, which is something I'd managed to keep clear of previously. It isn't as bad as I feared, and the nice haskell tls and x509 modules make it relatively nice to deal with, but I'll be happy not to get into the detail again.

Beyond that, there was the slow process of implementing commands which I realised should exist. I've consistently dogfooded throughout this process, and added features as I found I wanted them. Soon enough, two and a half months had passed and those 162 lines had ballooned to 2715.

Development was mostly rather smooth, but there were a few difficulties and questionable decisions which might be worth discussing.

Firstly, I struggled with Haskell's "configurations problem" -- because global variables are a big no-no in a pure functional language, it's difficult to have configuration options which will take effect at various points deep within the logic. Most non-hacky solutions to this involve explicitly threading the information through to where it's needed, which I find too ugly to contemplate. In the end I ended up using a method with its own problems -- just putting all the logic which needs to be configured within the scope of the configuration variables. This means it all has to go in one module, which isn't very nice. That's why there are two rather huge (by haskell standards: 658 and 358 lines) modules. Probably this was not the best choice, but maybe it was.

Secondly, I struggled a little with lazy IO. This is a feature of Haskell which the experts strongly advise not be used, and they have good reasons. But I used it anyway. Originally diohsc slurped an entire response before doing anything with it; once people started experimenting with streams, the foolishness of this approach became clear. So how to adapt code which operates on a complete response to handle it being streamed in chunk by chunk? Lazy IO provided a very tempting answer: the IO code receiving the response from the server can be adapted, without even changing its type signature, to return a lazy data structure representing the response. This appears to other functions to be a normal response, except that if they try to process it in a way which requires parts of the response which haven't yet been received, they will block until that part is received (or there's an error). I succumbed. It took a fair few iterations, and a few near-surrenders, before I got it working truly robustly, but I'm pleased with the final form. This has concrete consequences: if what the user does with a response requires only an initial portion -- e.g. because they ask only for the header, or they pipe it to head, or they quit some way through paging -- the connection will be automatically killed. All this could I'm sure be achieved with a more "modern" approach using the 'pipes' or 'conduit' haskell packages, but this would have "infected" a large part of the code and made it harder to read. I can see that there's something icky about pure code being less pure than it looks... but in this instance it seems pretty harmless.

Thirdly, more generally, I avoided "advanced" language features such as lenses or deep monad transformer stacks, as well as all the clever things I don't even know the names of. This may have been a mistake, lenses at least would have made many expressions decidedly neater, but hopefully it means the code will remain reasonably readable (to me, in particular) indefinitely. It isn't pure Haskell 98 by any means, and e.g. LambdaCase and OverloadedStrings would be painful to live without, but it's basically bog-standard haskell. I like bog-standard haskell.

The result

For a long time, diohsc was essentially an AV-98 clone. This wasn't the intention, and in fact I've played with av98 only briefly and occasionally, but convergent evolution and/or subconscious imitation meant they felt rather similar. The radical overhaul of the command system which I introduced in my previous post, which has only deepened since, changed this. I think diohsc now has a rather different feel from av98, or any of the other clients I've seen. It's been fun to explore the possibilities of diohsc's new command language. I'm finding it a nice mix of the keystroke efficiency of a curses interface and the powerful composability of a shell. I don't know how learnable it is, but hopefully adequately.

Meanwhile diohsc has everything I believe a unix program should have -- solid mechanisms for calling other programs, good behaviour in a pipeline, easily edited configuration and state, small memory footprint, low resource usage. My only complaint is the huge size of the binary itself, which scrapes 10MB when compiled statically, but it seems I can't do much about that.

And I think I should stop writing now. 0.1 should be released soon, i.e. I'll put it on hackage and make available some statically linked linux binaries (albeit of dubious portability), and announce to the mailing list.