My experience writing a Lisp system shell

A couple years ago I found a document that I can only describe as half academic whitepaper, half manifesto. It was dated to the early 2000s (decade), and discussed-then-evangelized the idea of replacing the Unix shell with a dialect of Scheme. The first half of the document, which was about 100 pages or so, was dedicated to enumerating problems with the Unix shell; many of the problems were related to long-term maintenance of software written in Unix shell languages. The document also discussed tradeoffs between interactibility and elements of strutured programming, which make long-term maintenance easier.

As a lisper myself, I was smitten by this idea. The author's own implementation of a scheme shell does not appear to have been very successful even at the time, and is not easy to compile with modern tooling. I set out to write my own version, which is the origin of the multi-phase "lish" project.

Phase 0 was trying to hack together a fast subshell system into Guile Scheme, which I never got around to writing, but did produce a bridge layer for Scheme and Rust:

rust-guile, on crates.io

Phase 1 was more straightforward. I wanted to write a new interpreter that implemented a subset of the Common Lisp grammar with 1 addition, for writing system commands as barewords.

The one thing that shell programs really get right that nothing else does is brevity. If I know that I want to run 8 or 9 system commands in a certain order or under certain conditions, the tradeoff is basically: the Unix shell will eliminate all boilerplate for running those commands, while dedicated programming languages will eliminate much of the boilerplate for doing anything else. Very often, this is a tradeoff worth making for shell scripts. The sysadmin rule-of-thumb that once your shell script reaches 100 lines you should rewrite it in a scripting languag is really downstream of this tradeoff, as it's implying that by the time you reach 100 lines, you have enough shell script boilerplate for things that aren't running commands (such as parsing command-line options, conditionals, loops, state management, etc) that you would save more by reducing the "other stuff" boilerplate than you would lose by increasing the "run commands" boilerplate.

High quality Lisps go to the extreme of reducing "other stuff" boilerplate, as any dedicated lisper will be able to tell you, so it seems intuitively like there's a lot of value-added to be had by either reducing the "run commands" boilerplate within Common Lisp specifically or by trying to add Lisp semantics to a Unix shell.

So, I started writing a language called lish, which was basically the former except a greenfield project. I got decently far, but eventually got bogged down by IRL things and stepped away from the project long enough that my hand-written docs no longer made sense to me. I considered rebuilding the project from scratch again based on what notes I could still comprehend, but chose not to. You can still find the source code for this phase of the project, although I am not sure if it will compile in recent versions of Rust.

lish, on GitLab.com

Instead, I started speccing out a "phase 2" for the project: the significantly less ambitious goal of creating a macro package for Common Lisp that would bring barewords system command execution directly into the Common Lisp REPL. By limiting compatibility to just SBCL and its native extensions, this should not be that difficult.

I have a document somewhere detailing the concept behind this. By far the most nuanced part of this undertaking is what to do about error-handling. Here's an analogy. The Ruby on Rails web framework abuses the hell out of a feature of the Ruby language called "method_missing", which basically is a callback the interpreter runs when a program attempts to execute code that the interpreter doesn't know about. Rails rewrites this callback to be a whole program that searches your codebase for files named in very specific formats, defining classes in a very specific way, and then runs code from those classes. This is how the "magic" of Rails's signature convention-over-configuration style of application structure works at the programming language level.

You could probably do something conceptually similar within Common Lisp to get a "fallback to shell" type scenario, but the ramifcations of that can be scary. In Rails, it is often completely unintelligible when you have a logic error in your application code that results in the method_missing program running unrelated code you weren't even thinking about. We probably don't want that in Lisp, which means finding a way to "guard" system commands behind a syntactic feature which will inherently make the system more boilerplate-y than a Unix shell, or...something else? Maybe introduce modality to the language itself, so you can be in command mode or program mode? But of course, that breaks the Lisp paradigm itself of making code and data the same structure.

Eshell solves this problem by allowing itself to be a really awful programming environment because, hey, this is eshell, you'll get a better programming environment by just writing Lisp in a file and executing it. I don't think that's an acceptable tradeoff for a dedicated Lisp shell, because then what is the point?

In case it isn't obvious by now, I never reached the point of writing a single line of code for lish phase 2. If there's any other common lisp enthusiasts out there who have a novel take on this design problem, I would love to see it, because I'm sure I'm just not being creative enough about this.

In fact, my attempt to spec out a lisp-plus-shell system has kinda made me think that the other way of shell-plus-lisp was really the way to go. Maybe we really just need advanced, succinct control flow and program structure capabilities in an environment that you otherwise interact with just like current Unix system shells. So, for phase 3, maybe I'll fork the Almquist shell and tack on just 1 more element to the grammer: S-expressions, containing the entire Common Lisp language within them :D