Why a New Protocol

I originally wasn't going to create any new protocols that mimic much of the same functionality as Gemini or Titan or Gopher. What changed my mind then?

The Scroll Protocol takes some inspiration from Gopher+. I read mozz's journal entry about Gopher+ recently:

Report: State of Gopher+, 2023

This led me to reading the Gopher+ spec:

Gopher+ Spec

And while reading this, even though there were a lot of convoluted things that Scroll deliberately does not take from Gopher+ (a scripts section?! yuck!!), there were a couple of interesting things that jumped out as being useful, particularly the metadata requests and the concept of abstracts.

In Gemini there is a lack of semantic detail. Gemtext files don't have a standardized way of getting the publication date or modification date or even an author metadata field. However, we got around this by using gemsub feeds, but it is very often that I come across gemlog pages where the publication date is on the gemsub feed but *not* in the gemlog page itself! And what about other filetypes that also don't include this metadata information, particularly text files? What if we just added these three extremely common metadata fields to all responses?

Gopher+'s Metadata requests are really interesting because they allow you to just get the metadata of a resource without actually getting that resource. This is actually really useful for browsers displaying information *before* getting the whole resource. This is even particularly useful for Search Engines and Crawlers to just check the existance of a resource, or whether it has been updated even, or to get its content type, before getting the actual file.

Gopher+ also adds in these "abstracts" to the Metadata requests. Abstracts offer a description of the file. This lets someone get a brief description of a file before spending resources downloading it. Abstracts in Scroll also offer the title of the resource so browsers and crawlers can get the title before downloading the resource. One use of an abstract could be to list the currently playing song of an audio stream within the abstract, and AuraGem's public radio does just this.

All of the above, however, introduces a new concept: a distinction between contextual information and non-contextual information. When you link to a resource from a page, you title the link in a way that it's contextual to that page. However, resource metadata is non-contextual; it doesn't depend on the context of another page.

Both contextual and non-contextual information is useful, and this to me justifies the use of documents like in Gemini over file listings like in Gopher. Documents give information about files within the context of the rest of the document. File listings presumably give non-contextual information. However, gopher already blurs this boundary by allowing you to link to outside resources from a menu! And thus, links that should be non-contextual, like in a filesystem, become contextual as file links are placed in groups, categorized, and even duplicated among multiple menus. And yet gopher's "i" itemtype that would explain this context, that would explain how a link relates to the rest of the links in the menu, is non-standard and frowned upon. Gemini makes the correct decision in allowing the already-blurred lines to be clarified by contextualizing links with text. Gemtext documents are better menus than gopher menus ever were for this simple reason: groupings of links (menus) can be contextualized and explained with text.

Mozz had it only partially right when he talked about the distinction of Documents and Directories in 2019:

Documents and Directories

It is true that Gopher does distinguish between menus and documents, whereas the web does not. In the modern web, every page has become a document to the point that directories hardly even exist anymore. Gopher has "directories" that are more like groupings of links, aka. menus. Gemini ends up using the same markup for directories and documents; however, the distinction is still present: a gemtext directory is usually constructed differently than a gemtext document. There are some cases, however, where the lines are becoming a bit more blurred like they became in the web, and this mainly comes from the social media and other interactive capsules. Scroll takes much the same approach as Gemini, but it tries to reintroduce the distinction between directories and documents through convention (see the "Structure of Content" section in the spec), particularly the convention of what is expected of super-directories in URLs.

Contextual information lends itself towards semantic information when linking between pages. This is one of the few things that search engines really need but that nobody has effectively created a solution for: how do search engines know when a link to a page implies a positive or negative relationship? Search Engines that use pagerank currently assume all links imply a positive relationship. Scrolltext adds a very simple tag to the end of all links to identify these relationships. It is done in a human-readable way so that they are useful to users, and so that they don't distract too much when the browser doesn't support these tags.

Lastly, and the biggest addition to scroll that I think justifies and warrants its existance, is internationalization, which Gemini and Gopher sorely lack. Gemini offers a lang parameter on responses, and allows UTF-8, which is already more than Gopher does, but there is no way to *request* resources of a particular language without creating a whole different link to the resource adding some language tag to the beginning of the link's path. Scroll solves this by adding a field to all requests for the languages the user desires, and then the server responds with a match for one of the languages, or a fallback if no matches.

I chose this method because the language should not be in links but be requested by the user's client. Links assume everyone reads in the same language as the one who posted the link. Rather, it should be the client, not the content, that decides what language is requested, *and* clients can offer up multiple languages in the request instead of just one language if the user speaks multiple languages. This is such an easy addition that I think will vastly improve internationalization. It also means all error messages may be internationalized by the server!

For all of the reasons above, I felt that these additions were very simple additions, and yet powerful enough, to warrant the creation of a new protocol.