Software Dependencies

2021-04-06

I found a neat program recently that I was interested in understanding. It happened that the project was written in Rust (which I don't know) but had good-enough documentation. I didn't have a Rust toolchain or much interest in Rust beyond this point in time but I decided it would be an interesting hobby project to try re-implementing it in a different programming language.

It isn't really important what the project is, or the language I ended up using because what really set me to writing was the horrifying realization after I had finished. I was interested in some samples provided with the original project and wanted to ensure my own implementation conformed; with the exact same outputs provided the same inputs. This necessitated I actually install Rust to build the project.

Cargo is the Rust build tool and (apparently) automatically fetches dependencies and compiles based on (I think) the `cargo.toml` file. I hadn't realized that there /were any/ dependencies but watched, first in amusement, then in horror, as library after library was pulled in. There were enough that I settled on cloning the repositories for each dependency to figure out just how much was there. I've done this in the past (with mixed success) at work to argue against dependencies but mostly I figure people should do whatever they want in their own projects.

While the original project and my re-implementation were approximately the same size, one required 36 supporting libraries and the other required none. The 36 Rust libraries end up being about *380,000* lines of supporting code.I wasn't drawing on some expansive standard library or leveraging functionality unavailable to Rust, I just happened to write only exactly what I needed. Just thinking about it drives me up a wall. In both situations the same input results in the same output but one requires hundreds of thousands of lines of libraries — for what‽

While I ranted a bit about NPM previously (and expressed skepticism about deno solving the problem) I had no idea that the attitude had spread quite so far. Stop pulling in so many libraries! What is going on?

As a nice bonus, when I re-wrote it myself the resulting program ends up being over 2,000 times faster. This is with no algorithmic wizardry or special tools. I used a dynamic language with garbage-collection and relied on blog posts I found on the internet to guide me in designing the thing. This is my argument for just doing things yourself: you can understand the whole thing and it can be faster! There is simply no way anyone understands all of the supporting code in a project with hundreds of thousands of lines of dependencies; how could you? That much code represents decades of work.

I don't purport to have any real skill at programming. I don't want to give an air of "look at me, genius programmer" — I wanted to re-write this as an exercise in trying a language I didn't know in order to learn something. If I can do as much then anybody smart enough to figure out Rust can too.

What I really want is for people to try the same thing themselves. What if we just, didn't, with all of this? What if you just wrote only the thing you needed every single time.