Gopher, Gemini, and the rise of the small Internet

Beyond the Web

Author(s): Lee Phillips

The danger and irritations of the modern web have unleashed a movement dedicated to creating a safer and simpler alternative. The old Gopher network and the new Gemini protocol have emerged as building blocks for this new "small Internet."

Anyone who has used the World Wide Web (WWW) lately knows that something bad is happening to it. It does not resemble the WWW of the early years, with enthusiastic amateurs freely sharing ideas and information. These things still exist, and the web is still an indispensable medium connecting the world. But the web experience is now encumbered with advertising, invasions of privacy in the form of pervasive tracking, enormous file sizes, CPU straining JavaScript, the danger of exploits, and door slams asking you to subscribe to a newsletter before viewing a site.

This unpleasant environment has led to a backlash. There are now some communities of developers and computer users who still desire a connected information system, but who seek a refuge from the noise, danger, and increasingly resource-hungry WWW. They feel that web technology does too much, and that since it makes various forms of abuse too easy, no lasting reform is possible.

The solution is to use or create a separate protocol that is simply not capable of supporting the technologies that enable advertising networks, user fingerprinting, or the myriad of other things that exploit users rather than helping them. This small movement has approached the problem from two directions that in practice are often merged: the revival of the Gopher protocol and the creation of a new protocol called Gemini.

A Little History

Looking back at computer networks before the emergence of the Internet in the late 1980s, and even the Internet itself in its early years, there were only a few applications of interest to most users: email, logging on to timesharing systems, and the transfer of documents through the file transfer protocol (FTP).

FTP began as a command-line program, with graphical clients eventually arriving for every platform. To get a file, you had to know its specific address on the FTP server where it resided. Eventually, it got a bit easier to find things with the arrival in 1990 of Archie, the first public search engine on the Internet, which indexed FTP files.

A command in the terminal would download the file to your machine. You could navigate through directories if they were exposed on the server, but you could not navigate outside the server, discover content, or interact with the files before downloading them. FTP was essentially a networked version of the cp command. It was, for years, the only way to share content, aside from passing around tapes and floppy disks; FTP was the way the first versions of the Linux kernel were distributed [1].

Shortly after the Internet's predecessors evolved into the modern global network, other means of sharing content emerged that offered more than FTP's spare file copying. The three most significant participants in this information landscape were the Wide Area Information Server (WAIS), Gopher, and the WWW.

WAIS was a sophisticated software system [2] that translated the user's natural language query into search commands for databases. From a networked computer, you could search the contents of databases all over the world and retrieve a variety of documents. There were textual and graphical clients for major operating systems, with varying degrees of polish.

WAIS was developed by Thinking Machines Corporation [3], a visionary company that created the first massively parallel supercomputers. WAIS is completely dead now. Even if a client could be found that ran under the current version of any OS, there is no content and no servers to interact with.

Gopher, on the other hand, is still being used. As Cameron Kaiser points out [4], "Gopher takes the strict hierarchical nature of a file tree or FTP and turns it into a friendlier format that still gives the fast and predictable responses that they would get by simply browsing their hard drive."

And that serves to explain the nature of Gopher: Like FTP, it was a way to display directories of files that could be downloaded, or in the case of text files, displayed directly by the client.

Gopher clients went beyond simply downloading files, because they had some notion of file type, and could respond appropriately, in a limited way. While FTP servers only presented lists of files and directories local to that server, Gophermasters could create directories that appeared as pages of links and text. The links could lead to other servers, creating the first easily accessible global network of information. These directories were not hypertext documents, however; they contained isolated links and descriptions of content.

Around 1990-91, the first WWW browsers appeared on computers all over the world, and in another year or two the web had surpassed FTP, WAIS, and Gopher as the medium for finding information and entertainment through the Internet.

At that time, I was working in a government laboratory, enjoying the luxury of the kind of fast Internet connection that most of the developed world takes for granted now. I had several FTP clients and at least one Gopher and WAIS client on my laboratory computer. The reasons for the rapid ascendency of the web are not mysterious to me, because I distinctly remember watching it happen in real time.

As soon as people saw hypertext, it captured their imaginations. That is the magic of the web: to click on a word in a document and be taken somewhere else for further enlightenment. And these were not merely text documents; they could be styled to look like magazines. When inline images and other media appeared, it sealed the deal. In the lab, we started making our own pages to share information with other departments. This was the original purpose of the web, for scientists to share research with each other, and in our lab it was a reality.

Everyone I knew with a computer, including laypeople who had never heard of FTP or WAIS, was soon browsing the web. The WWW had replaced, more or less, the existing protocols and became the standard way to share information.

In 1992-93, as the WWW was just beginning to surpass WAIS and Gopher in network traffic, the WWW's inventor, Tim Berners-Lee, wrote a note explaining the differences among the three systems [5]. Berners-Lee pointed out the advantages of hypertext over the other methods of distributing information and emphasized an additional crucial virtue of the WWW – it could be used as an interface to Gopher or WAIS and was therefore a superset containing the other two protocols. Thus the web rendered Gopher and WAIS obsolete in a technical sense in addition to replacing them as a preference on the part of users. Fast-forward to today, and even Tim Berners-Lee thinks the web has gone too far. Berners-Lee is one of many experts who have spoken out against the perverse incentives and lack of privacy that characterize today's web [6], and he has even launched an initiative to try to fix it [7].

The first murmurings of a revival of old protocols as reaction to the excesses of the web mentioned Gopher [8] as a plausible candidate for a refuge from the storm. (See the box entitled "URLs and Protocols.") Some advocates for the Gopher revival speak of a "community outside the chaos of the modern Internet."

URLs and Protocols

The familiar URL, like https://google.com, contains several parts. The part before the colon is the protocol, in this case secure HTTP. The new protocols that go beyond the web borrow this URL structure. Gopher addresses look like http://gopher://example.com, Gemini locations like http://gemini://example.com, and FTP sites like ftp://example.com. You can configure your browser to open each protocol with a specific application.

This is a relatively easy way to build an alternative Internet community, because the Gopher universe never completely disappeared. There are clients that still work, or can be updated to work, on current operating systems.

Now Gopher is growing again, specifically because it is not the web. The number of Gopher servers is again increasing, a development that would have seemed highly unlikely as recently as five years ago. Because it does not support any kind of scripting, much of the abuse of the modern web can not occur. The lack of embedded media removes the possibility of most forms of advertising or other irritations. And its very retro nature gives the use of Gopher an inherent cool quality.

Enter Gemini

Project Gemini [9] has no interest in replacing either the web or Gopher but seeks to live alongside them as another option for disseminating and consuming content. It carefully positions itself between Gopher and the WWW in its complexity and capabilities. The creators of Gemini realized that Gopher lacks certain features that are desirable for dissemination of even simple documents over the web, such as encryption and the inline inclusion of images along with text. But they deliberately excluded from the specification [10] web features such as scripting and complex document models that are more likely to enable abuse of the information consumer. To this end the protocol is designed to be difficult to extend, so that its protections against abuse will endure. One example of Gemini's emphasis on privacy is that connections are required to be encrypted. Another is that the protocol has no provision for User-Agent or Referer headers, and it is specifically designed not to be extensible to include them.

It has several advantages over the older Gopher protocol, chiefly in the use of non-ASCII character sets, using MIME types to handle non-text content, redirects, and virtual hosting; the requirement to use TLS encryption; and links within documents.

The type of link implemented by Gemini is not freely mixed in with the document as in HTML; instead it sits on a separate line. The Gemini designers see this as an advantage, as it helps in the organization and outlining of documents. We'll take a closer look at the native Gemini file format in Listing 1, when we try out one of the Gemini browsers.

Listing 1

A Gemtext Example

GUS - Gemini Universal Search

Home

Search GUS

Query backlinks

Search

Enter verbose mode

"cornea"

gemini://gemini.circumlunar.space/

users/lel/blog/anp/q4/ gemini.

circumlunar.space/users/lel/blog/anp/q4/

(text/gemini, 14K)

gemini://gemini.circumlunar.space/

~lel/blog/anp/q4/ gemini.circumlunar.

space/~lel/blog/anp/q4/

(text/gemini, 14K)

Page 1 of 1 (2 results)

> "Nobody ever figures out what life is

all about, and it doesn't matter. Explore

the world. Nearly everything is really

interesting if you go into it deeply

enough." --- Richard Feynman

See any missing results?

Let GUS know your Gemini URL exists.

Index updated on: 2020-12-06

This article will not go into setting up content servers, but those interested in serving Gemini content will find some free, hosted avenues for doing so at the Gemini FAQ.

The Gemini space is still small, but growing rapidly. For historical interest, the project exposes a list of the first 50 known Gemini hosts at (naturally) a Gemini address. Up-to-date information, including an automatically generated list of Gemini hosts, is available at the Gemini Universal Search (GUS) search engine.

Still, there are, according to GUS, about 200,000 Gemini pages served from 422 domains, compared with over a billion pages on the World Wide Web. Consequently, Gemini is not a place to find information or do research, nor even to browse interesting content in depth. Today it is still a place to explore a new idea in information sharing and perhaps to enjoy wandering through a small but enthusiastic networked community while knowing that your privacy and security are being respected and without the irritations of the modern web. Gemini fills a niche, is well executed, and has been thoughtfully designed. Its limitations are very much deliberate choices. I believe it will continue to grow and stands a reasonable chance of becoming a real second place to find and share information on the Internet.

The new Gemini protocol, along with the revival of Gopher, and the attitudes and community that go with them, are sometimes called the "small Internet," in contrast to the WWW, which is obviously anything but small. This nascent movement also embraces the venerable Unix Finger protocol. The finger command in Unix machines returns a small amount of text indicating the status of a user or a machine. Despite its simplicity, it has been used by creative individuals as a minimalist communications medium.

Part of the appeal of Gemini is the same thing that makes Gopher attractive to some people: The very fact that it is still small and new makes browsing the available content seem like exploring a new universe, somewhat like the feeling of exploring the web in the early years.

Linux Gopher Clients

VF-1's motto [11] is "High speed, low drag." It is a basic terminal client, similar in operation to the command-line FTP clients of olden times. VF-1 is written in Python and can be installed with:

<C>pip3 install VF-1<C>

After the user types vf1 in a terminal, a brief welcome and a simple prompt appear. If the user guesses correctly and types help, he or she will be rewarded with a list of the available commands (Figure 1). One of these is veronica, which submits a search term to the Gopher search engine of that name. Navigating to results is accomplished by typing the number displayed next to the file name and hitting return, after which the file contents are dumped to the screen. Alternatively, the user can view the file in the less pager or save it to disk.

Figure 1: A list of the available commands for the VF-1 Gopher client.

If you know where you're going, the go command will take you right there. In Figure 2, I've used this command to navigate to a Gopher mirror of the Hacker News front page. Gophermasters often adorn their directory pages with ASCII art to personalize them in the absence of inline images.

Figure 2: The Hacker News Gopher mirror in VF-1.

The next step up in convenience from a pure command-line program like VF-1 is a curses-style interface. These programs are still lean and fast, and they run in the terminal, but through random access to the screen area they allow a richer mode of interaction and display.

Phetch is a Gopher client in this mode written by Chris West. Binaries and source are available on GitHub [12]. After typing phetch in the terminal, the program presents the startup screen shown in Figure 3. Navigation is accomplished through the keyboard, with links and text being distinguished by color. As this is Gopher, text is all there is, so the terminal remains a reasonable environment for consuming the content, as shown in Figure 4. Phetch does what it was designed to do with an intuitive, efficient interface. I found it a pleasant way to explore the Gophersphere and encountered no bugs or rough edges of consequence.

Figure 3: The startup screen for phetch, a Gopher client that runs in the terminal.

Figure 4: A page from the Gopherpedia, the Gopher interface to Wikipedia, shown in phetch.

There appear to be no existing graphical Gopher clients for Linux that are not broken or too ridden with bugs to be useful.

FORG [13] is a graphical Gopher client written in Python. The GUI uses tkinter. I was able to start FORG by executing the main Python file after downloading the repository from GitHub, but first I had to use pip to install a dependency, Pmw. When I tried to use it, I found that FORG displayed its configured home page with no problem, but any link I followed resulted in a blank page and error messages written to the console. I hope that this program gets these issues resolved, because it has some valuable features.

Another GUI client too buggy to use in my testing on Linux, but apparently popular on macOS, is called Little Gopher Client.

Finally, some web browsers, including text-mode browsers such as Lynx and ELinks, have support for the Gopher protocol, so it's likely you can browse Gopherspace without downloading any new software. There is also a growing number of Gopher extensions for browsers that don't come with support out of the box.

Linux Gemini Clients

Jaakko Keränen's Lagrange [14] is a highly polished GUI Gemini client. One minor warning: On high-resolution screens, Lagrange's user interface will be unusably small. Fortunately, there is a preference item under Window called UI scale factor that works perfectly, although the program needs to be restarted before it takes effect.

There was only one other tiny glitch: I have my Caps Lock key mapped as a second control key, which works in every other application, but is not recognized in Lagrange. However, there is an easy-to-use preference item for remapping the keyboard shortcuts, and with this it only took a few seconds to fix this glitch for most shortcuts. Regrettably, certain others can not be remapped; so I still need to use the pointing device to enter an address, for example.

Lagrange embeds images linked from a page into the display, rather than opening a new window. An important part of the Gemini protocol is that the clients will not make network requests that are not manually initiated by the user, so normally the reader will click on each image of interest. However, Lagrange offers a preference setting that loads images when they are scrolled into view, reproducing an experience more like a conventional web browser. With this setting selected, images are inlined when scrolling with the arrow key or the spacebar, but the user can use the scrollbar to scroll without triggering image loading. This is another example of Lagrange's pleasant and well thought out user experience. Lagrange can also embed an audio player in the displayed page.

Lagrange can display a navigable outline of the page in a sidebar, but it requires the mouse to jump between page sections. The two sidebars can also display other information, such as bookmarks, history, and TLS client certificates. This mature browser also lets you search for text on the page. It is quite configurable through its preferences dialog, allowing the user to choose or reject smooth scrolling and set many aspects of the appearance and user interface. It has tabs as well.

Lagrange has keyboard navigation that works like link hinting in Vimperator and similar systems, but it is even easier to use. It caches recent pages, so going back and forth is instantaneous.

In Figure 5, you can see (both) results displayed in Lagrange from a search for "cornea" using GUS. The rendering shows most of the common elements that a user may encounter on a Gemini page: regular text, two different levels of headings, links both within the server and to pages on other servers, and a quotation. It is up to the client to decide how to display these various elements. Lagrange distinguishes server-local and remote links with a triangle or a globe, shows link destinations when hovering, shows you what links you have visited by coloring them differently, displays link hints when hitting F that replace the triangle or globe symbols, and has special formatting for quotations.

Figure 5: The results from searching "cornea" in the Gemini Universal Search, as shown in the Lagrange Gemini browser.

Let's take a look now at the markup language used by Gemini. The source for the page in Figure 5 is shown in Listing 1. Readers familiar with Markdown will recognize the inspiration for the syntax, but this markup format, called "Gemtext," has some important differences from Markdown and from HTML.

The most unusual aspect to Gemtext markup is that it is based on single lines, with each line possessing one type. Paragraphs of normal text exist as single long lines, wrapped by the client for display. The longer lines are wrapped here to fit the format of the page. The other major line types are signalled by the characters at the start of the line. Headers are indicated by # characters, links by =>, and quotations by >. By design, links are on their own lines and can not be intermixed with text, as mentioned above. Gemtext therefore is deliberately designed to not be a full hypertext format.

The line-oriented specification has two purposes: to create a well-structured, organized document, and to make it as easy as possible to write parsers for the format, making it more likely that clients will proliferate. It is possible to write a Gemini client in under 100 lines of code in a high-level language like Python.

Amfora [15] is a browser exclusively for Gemini, like Lagrange, but running in the terminal rather than in a GUI. It is distributed as a simple binary for Linux, all BSDs, macOS, and Windows, as well as, of course, source code. Since Amfora is navigated solely through the keyboard, some form of link hinting is required. As shown in Figure 6, Amfora supplies numerals beside each link in the document display. To follow a single-digit hint, the user need merely type the digit; to disambiguate multiple-digit hints, hitting the spacebar takes the user to the address area at the bottom. This system of navigation is quick and efficient. Amfora does not display images inline, as might be expected. Selecting an image will give the user a choice to download it or display it in the default external application. Amfora has tabs, handled neatly in the text interface, as shown in Figure 7. I could find no way to search within a page, however, which is a serious omission.

Figure 6: Information about Project Gemini, displayed in the Amfora Gemini browser.

Figure 7: The Amfora browser, which runs in the terminal, offers some interesting features, including the tabs shown here.

In my trial of Amfora, I found that it had no problem following links and quickly navigating backward and forward, but that there was an issue in handling forms, as queries to the GUS search engine became stuck and never returned results. This was not a problem with the server, as identical queries at the same time from Lagrange returned results almost instantly. However, the problem only exists when trying to use the dialog box that pops up when querying GUS through its search link. If you enter Amfora's command area by hitting the spacebar, you can enter search terms there, and pressing the return key sends them directly to GUS as URL parameters and quickly displays the returned results.

Finally, if you would like to browse Geminispace content without installing one of the small Internet browsers, there is a portal for the WWW [16] that works in a normal web client.

Combined Clients

Several clients exist that can handle both Gopher and Gemini (and sometimes Finger, as well).

One of these is called Bombadillo [17], a terminal program that can handle all three protocols, plus local files. Bombadillo can be compiled from source on any Unix-like operating system if you have a Go compiler installed, and there are precompiled binaries available for several processor architectures on Linux and for macOS.

Figure 8 shows how Bombadillo renders the GUS search results page that we saw Lagrange handling above. It uses numerals for navigation, using the same system as Amfora for following links with either one digit or more. Unlike Amfora, Bombadillo does not color links, and, as Figure 8 shows, does not break lines correctly, splitting words at arbitrary places. Also unlike Amfora, Bombadillo handled the query to the search engine with no problem, using a one-line command area at the bottom of the window to accept the search terms. Like the other clients reviewed here, Bombadillo has back and forward navigation and a way to save the user's bookmarks. Operation is quick and painless.

Figure 8: Bombadillo's rendering of a GUS search results page.

Another graphical client supporting Gemini, Gopher, and Finger is called Castor [18]. It is written in Rust using the GTK GUI library.

Have Fun!

The new phenomenon of the "small Internet" is still not a place to go for serious research. But it is, today, an interesting place to explore. After the cacophony of the modern web, browsing through Gopher or Gemini land feels like a quiet stroll on a remote beach, where you might just find the occasional beautiful shell lying on the sand.

There are good reasons to expect Gemini, in particular, to evolve into a viable alternative to the modern web. For now, I intend to keep a couple of small Internet browsers installed and visit the beach on occasion.

Infos

"25 Years of Linux" by Brian Proffitt, Red Hat Blog, 2016: https://www.redhat.com/en/blog/25-years-linux

WAIS: https://www.ou.edu/research/electron/internet/wais-faq.htm

"The Rise and Fall of Thinking Machines" by Gary A. Taubes, Inc.com, 2020: https://www.inc.com/magazine/19950915/2622.html

"Why Is Gopher Still Relevant?" by Cameron Kaiser: https://gopher.floodgap.com/overbite/relevance.html

"W3 vs. WAIS and Gopher" by Tim Berners-Lee, W3.org, 1992: https://www.w3.org/History/1992/WWW/FAQ/WAISandGopher.html

"30 Years On, What's Next #ForTheWeb": https://webfoundation.org/2019/03/web-birthday-30/

Contract for the Web: https://contractfortheweb.org/

"The Web May Have Won, but Gopher Tunnels On" by Nate Anderson, Ars Technica, 2009: https://arstechnica.com/tech-policy/2009/11/the-web-may-have-won-but-gopher-tunnels-on/

Project Gemini: https://gemini.circumlunar.space/

Gemini protocol specification: https://gemini.circumlunar.space/docs/specification.html

VF-1 command line Gopher client: https://github.com/solderpunk/VF-1

phetch: https://github.com/xvxx/phetch/releases/tag/v1.0.0

FORG on GitHub: https://github.com/Tom4hawk/FORG

Lagrange Gemini client: https://git.skyjake.fi/skyjake/lagrange

Amfora terminal browser for Gemini: https://github.com/makeworld-the-better-one/amfora

Gemini portal: https://portal.mozz.us/gemini/gemini.circumlunar.space/

Bombadillo: https://bombadillo.colorfield.space/

Castor browser for Gemini, Gopher, and Finger: https://sr.ht/julienxx/Castor/

The Author

Dr. Lee Phillips is a theoretical physicist and writer who has worked on projects for the Navy, NASA, and DOE on laser fusion, fluid flow, plasma physics, and scientific computation. He has written numerous popular science and computing articles, as well as technical publications, and he is engaged in science education and outreach.

In the US and Canada, Linux Magazine is known as Linux Pro Magazine.