Gopher, the Father of the Internet

August 4, 2019

This article pays tribute to a pretty cool protocol I came across whilst browsing a list of TCP/UDP ports on wikipedia, Gopher.

I thought this was a tech blog, why are you talking about animals?

The Gopher protocol was created at the University of Minnesota, by a team led by Mark P. McCahill. Minnesota is known is as the gopher state, so naturally a bunch of smart guys named a tech protocol after an animal.

Gopher is often called the predecessor of the web, and that's a pretty fair comparison given its use and capabilities it provided in the early 90's. The early 90's are a scary place for millennials like myself, there was no concept of "googling" the answer to that burning question you had, or using Stack Overflow to help solve that strange esoteric javascript error that according to your users only occurs when they double click 4 times, refresh, change tabs and then come back to your page.

Gopher provided early internet adopters with a way to share their ideas, and push their content out there on what were and still are known as GopherHoles.

Gopher is inherently a very simple protocol that wraps around TCP and uses plain text communication. If you're willing to stray away from what's defined in the RFC you can use gopher for an array of cool things! You could build a JSON viewer in gopher, a solitaire game, a sports score viewer, a tic tac toe game, it really is endless.

However, gopher was not designed with these capabilities in mind, gopher first and foremost was designed to be a distributed document delivery system. Now there's a few buzz words in that, but the core idea of gopher was it was to be used to present users with a hierarchical view of items and directories, much like a file system. Initially this enabled users on campus to share and access documents over a network, later this enabled internet users to share documents!

So, what does gopher look like?

At its core, a single line of gopher (also known as a Gophermap) looks a little like this: 1News /news gopher.koryporter.com 70 \r\n .

In the above example we can break those down into a few sections.

1 - directory item type
News - display string
/news - selector
gopher.koryporter.com - hostname
70 - port

Now there's a few things here that need to be called out. First, the delimiting character between the sections like 1News and /news, are tab characters, or \t. Secondly, you'll notice that the line ends with \r\n, the \r\n is a carriage return and a linefeed right after one another, it's more commonly known as a CRLF signal. Gopher browsers use this to delimit each line of gopher, if you had multiple lines you would need a CRLF on the end of each of them. Lastly, and most importantly, the single period . at the end of the line, that's the end of message character, it tells gopher clients that you have finished writing.

Check out this excerpt from the RFC for gopher, it goes through a typical connection scenario between client and server. Source: https://datatracker.ietf.org/doc/html/rfc1436

Client:   {Opens connection to rawBits.micro.umn.edu at port 70}
Server:   {Accepts connection but says nothing}
Client:   <CR><LF> {Sends an empty line: Meaning "list what you have"}
Server:   {Sends a series of lines, each ending with CR LF}
0About internet GopherFStuff:About usFrawBits.micro.umn.eduF70
1Around University of MinnesotaFZ,5692,AUMFunderdog.micro.umn.eduF70
1Microcomputer News & PricesFPrices/Fpserver.bookstore.umn.eduF70
1Courses, Schedules, CalendarsFFevents.ais.umn.eduF9120
1Student-Staff DirectoriesFFuinfo.ais.umn.eduF70
1Departmental PublicationsFStuff:DP:FrawBits.micro.umn.eduF70
          {.....etc.....}
.         {Period on a line by itself}
          {Server closes connection}

What can gopher do?

Let's first investigate what those characters are at the beginning of every line of text that gopher servers respond with, and why they are important for gopher clients.

Every single line of gopher usually begins with a character that exists between 0-9 or a-z, this is the reserved range of "Item type characters". The original gopher RFC defined just over a dozen item types for clients to respect, they are as following.

Item Type	Description
0	Text File
1	Gopher directory (submenu)
2	CCSO Nameserver
3	Error code returned by a Gopher server to indicate failure
4	BinHex-encoded file (primarily for Macintosh computers)
5	DOS file
6	UNIX uuencoded file
7	Gopher full-text search
8	Text-based Telnet session
9	Binary file
+	Mirror or alternate server (for load balancing)
g	GIF file
I	Image file
T	Telnet 3270

These types are used by gopher clients to understand what type of content it should expect at that directory.

See the following examples:

"1News /news gopher.koryporter.com 70" has an item type of 1, which means the gopher client should expect a directory (submenu), and that the directory will contain a listing of other gopher items.
"0my article /my-article gopher.koryporter.com 70" has an item type of 0, which means the gopher client should expect a text file, and that once the client has asked for the file, and the server has responded, it should then render out the contents of the text file in the client.

Now that we know gopher has item types, let break down a few key use cases for gopher.

Acting as remote file system viewer

As gopher clients can render text files and download binary files to a users system, we can use a gopher client as a very primitive way of viewing a directory structure on another system. This is what gopher was originally designed for really, it gave users a way to browse and access content remotely.

Full text document searcher

The gopher item type 7 defines a full-text search. Common practice among gopher clients is to give users the opportunity to enter any free text and it will send it to the gopher server. Full text searching means that the listening gopher server is meant to implement a search across the contents of every file it had available to it. The power of this comes into play when you can nest an item type of 7 deep down into a nested structure. An example might be a gopher server that displays books available. Instead of doing an expensive full-text search across all books, the gopher server might have you browse existing categories, and once you have selected your category, it would then present you with a full-text search (item type 7) within that category, which dependent on how many books and categories indexed by the server, would increase performance exponentially.

Simple Gopher Server in Node (optional)

If you whipped up a tcp server on port 70 right now, any gopher client would accept that as valid gopher text, in fact, let's do just that using Node.js.

const net = require("net");

net
  .createServer((socket) => {
    socket.once("data", (buffer) => {
      // sockets receive and stream buffers by default.
      const data = buffer.toString();
      // \r\n === CRLF, this is asking for the "root directory"
      if (dataAsString === "\r\n") {
        socket.write(`1News\t/news\tlocalhost\t70\t\r\n.`, () => socket.end());
      } else if (dataAsString === "/news\r\n") {
        socket.write(`iNice!\t\r\n.`, () => socket.end());
      }
    });
  })
  .listen(70);

Once you've run the above code a tcp server will be listening on port 70.

Open up your terminal and install the lynx web browser if you don't have it yet. Mac users can install lynx through brew by running:

brew install lynx

Otherwise checkout https://lynx.invisible-island.net/lynx2.8.9/index.html for more comprehensive instructions.

Now that you've got lynx installed, in the terminal run:

lynx gopher://localhost:70

Woop woop, you've created your first page in gopher! The output on your terminal should look a little like the picture below.

A mac OSX terminal output displaying Gopher — Woo-hoo, things are happening!

Thanks to the power of lynx, we can navigate through our little gopher hole using our arrow keys, because we only have one directory set up so far "News", that will be pre-selected for you. Click the right arrow on your keyboard. Lynx will then ask your gopher server for the contents at "/news", and you should be navigated to a new view that says "Nice!".

If you're interested how in the implementation of gopher using node, checkout this repo I created. A lot of abstractions were built to hide away the intricacies of gopher, see the core folder for more info.

So, you say gopher's distributed?

With that in mind, let's go back to the fact that gopher was designed to be a distributed system. In that single line of gopher you have just seen, you'll notice it has a URL and a port in it. You might rightly wonder why each and every line of gopher requires a URL and a port, shouldn't the client just remember that from the first request, seems like a waste to include it every single time?

Because every line of gopher in a listing has a server URL and a server port, we can actually create gopher servers that point users to other systems, without them really feeling like they have navigated away from the initial gopher hole they visited. At the time when gopher was in active use, this enabled campuses to essentially have one or more "root" servers, that held a source of truth of where other gopher servers existed, and what they were used for. Let's say that root server was really simple, when you request a directory listing from it (CRLF), it just returned a list of the different departments within the university (each with a different URL!), and with an empty selector string. From this point, a user could navigate to the department they were interested in, and the gopher server on the receiving end has no idea that the user actually came from a root server, it just sees a connection coming in asking for it's listing (CRLF).

Essentially, the ability to be able to point users to other gopher servers in such a manner meant that server administrators had an easy time creating redundant and highly available gopher servers.

Gopher + Javascript = 😎

Coming soon! Stay tuned for a detailed explanation on the abstractions that we can build on top of gopher using javascript to create some pretty cool applications! gopher.koryporter.com:70 has a neat JSON viewer in it, all you have to do is give a link to a JSON file, and it will do it's best to display and let your traverse through the contents of the JSON.

Thanks

If you've made it this far, thanks for reading! Gopher has become a little bit of a pet passion for myself, and if you have any feedback or just want to chat about gopher, feel free to send me an email, you can find it at the bottom of this page.

Kory Porter.