Build scripts for non-code projects

2023-01-18 | #tag | @Acidus

Build scripts are super helpful. By "build scripts" I mean Makefiles, a publish.sh shell script, really anything that automates building or creating something. Usually that's a program, but I find them helpful even for things that are not strictly speaking development work.

For example, I have been porting the content from some Apple history projects to Markdown. I also am publishing them as gemtext on my capsule. This is an iterative process, so generating the gemtext is something I do as I add more content.

Apple History and the Creation of the Macintosh

History of the Apple II

I know what you are thinking. Converting Markdown to gemtext? that's so easy! Just a run of md2gemini right? And that's why most people don't write a build script. They think its *just* one thing, or one command. But when you break it down, its actually more.

Lets see a non-code example. Those 2 content projects consist of Markdown documents and images, and the structure of the projects looks like this:

Posts/ - directory with Markdown files

Posts/images/ directory with all the images referenced by the Markdown in Posts/

publish.sh - my build script

Let's see how I have evolved the build scripts over time to convert the Markdown and images into gemtext for my capsule.

Clean output each time

First, I want to make sure that when I'm generating I'm not going to accidentally clobber anything. I should put the converted text into an output directory. I also don't want old or out-of-date copies to be there, so this should start clean and fresh each time. So my build script `publish.sh` looks like this:

# remove any existing output directory, and create a new clean one
rm -rf output
mkdir output
# copy all the Markdown, and the images/ directory into output/
cp -R Posts/* output/.

Now, all my actions will operate on the contents in output/, and I don't have to worry about messing up any of the source content.

Convert Markdown to gemtext

This is pretty straight forward. I'm running a forked version of md2gemini that allows me to convert image links in the Markdown to link lines in gemtext, but doesn't convert the hyperlinks in the Markdown.

md2gemini and Images

The command is:

md2gemini --write --dir output/ --frontmatter --img-tag='' --links images-only output/*.md

This command will generate .gmi files in output/ for all the Markdown files (*.md) in output/ as well. Of course, I don't need those md files anymore, so I can delete them:

md2gemini --write --dir output/ --frontmatter --img-tag='' --links images-only output/*.md
rm output/*.md

Unfortunately, md2gemini generates files that use CRLF to end each line, even if you run it on a system like Linux or macOS which just uses LF to end a line. If I edit these files, and end up with some lines ending with CRLF and some with LF, this can cause problems with some gemini clients or text editors. So I convert all the gemtext files in output/ to only LF for newlines:

md2gemini --write --dir output/ --frontmatter --img-tag='' --links images-only output/*.md
rm output/*.md
dos2unix -q output/*.gmi

Now I just have clean gemtext files (*.gmi) in output/ as well as images those files reference in output/images/.

Optimize images

When porting the source content to Markdown, I try and keep the highest quality and resolution images possible. This is because I'm trying to preserve this content before it rots away on the web. However a 1.5 MG JPEG of the cover of Time Magazine from the early 1980s isn't really appropriate for Geminispace. I should resize it. Perhaps I want to only have it be no more than 800 pixels wide or 800 pixels tall. I can use ImageMagick's `mogrify` command to bulk resize the images.

mogrify -strip -quality 80 -resize 800x800\> output/images/*

This also strips the meta data from the images, to make them smaller as well. Because I'm operating in the output directory, I can do anything I want without fear of messing up the source images.

Additional items

So far, my publish.sh looks like this:

# remove any existing output directory, and create a new clean one
rm -rf output
mkdir output
# copy all the Markdown, and the images/ directory into output/
cp -R Posts/* output/.

# Convert Markdown to gemtext
md2gemini --write --dir output/ --frontmatter --img-tag='' --links images-only output/*.md
rm output/*.md
dos2unix -q output/*.gmi

# optimize images
mogrify -strip -quality 80 -resize 800x800\> output/images/*

But because it's a script I can always add more things:

I have a C# program that creates an index of gemtext posts, and appends navigation links like "Next", "Previous", and "Story Index" to the end of each gemtext post

I have a few `grep` and `perl` calls to generate a consolidated image gallery file `gallery.gmi` of all the images in all the posts

I could add a command to generate a PDF, gempub, or epub file of the content

You can see the actual build script I use below (I happen to convert it to make, but it was a shell script)

Makefile for Folklore.org Mirror publishing

Freedom with Automation

This is how automation with a built script frees you, even for non-code projects. Before I was trying to save images and make sure they were the appropriate format for gemini. Now, I don't care. I don't need to worry about it. I have a system that will always copy and optimize the images, no matter what.

It was only after I had a built script that I realized how often I needed to do the steps. Since it was manual and ad-hoc before, I would do it more rarely, instead waiting for a large number of changes to "build up." With a built script I go so much faster. Fix a typo? Rebuild the content. Add a higher quality image? Rebuild the content. Once its automated, the cost to use it is effectively zero.

Also important is that consistency. Once its automated, the same thing happens every time, without you needing to remember the steps. And speaking of remembering, build scripts make it easy to return to a project later the future. I don't have to remember all the steps needed to publish my Apple II history project. I just run publish.sh.

But this project is "too small" to need one.

Everyone thinks the project they are working on is "too small" for a build script. But chances are, if you are working on a project, you have probably already written a build script without knowing it. You probably have some terminal open and you keep pressing "up" a few times to go back in your command history and re-run something. Maybe you run 1 command, and then run a second command over and over again as you are iterating on something trying to get it right. Just copy that command into a publish.sh file! In fact, if I ever find myself repeatedly needing to execute 1 or 2 commands over and over I just go ahead and make a shell script for it. These quickly evolve into publishing scripts.

-- Response ended

-- Page fetched on Tue May 21 17:31:04 2024