gemini://auragem.letz.dev/devlog/20240508.gmi

It said something that I found really interesting, mainly because I have a different look on things:

> Other Gemini search engines have gained popularity since then, from Kennedy to AuraGem. But Geminispace.info has always been my favorite, and Geminispace.info is the one I always return to.

What prompted me to write this article was the implication that AuraGem Search has gained popularity. As the person who wrote and manages AuraGem Search, I have a different perspective on this: AuraGem Search was an utter failure.

AuraGem Search's development has been riddled with issues that I believe hampered its growth, both in terms of features and popularity. I was originally prompted to create this search engine because Geminispace.info didn't cache page titles and so you couldn't see page titles in search results. It therefore was meant to be a vast improvement over GUS/Geminispace.info by caching more page and file details in general. I don't actually have a record of when I started developing it, since I didn't put it in a git repo early in its development, although I do know that I released it back in 2021 as "Ponix Search", Ponix being the name of this capsule at the time:

Ponix/AuraGem periodically went down in various times over the course of the last four years. Sometimes this was due to the capsule crashing and me never bringing it back up for months, and sometimes I just stopped development on it while focusing on college or other projects, and the last time was when the former domain (auragem.space) expired and I didn't have money to buy it again. These downtimes were prolonged, but I always ended up coming back to it. I have relaunched AuraGem at least 3 times in the past four years, and I've switched the domain at least 2 times as well, from pon.ix.tc to auragem.space to auragem.letz.dev.

AuraGem's search engine as it started was very slow to search because it didn't use Full-Text Search for any fields in the database. The search engine never cached the full text of any pages or files, but it had this tag system (i.e., keyword extraction) that would look for important words within the text and put those in the db to be searched on. Eventually this was removed becuase it slowed down searches way too much. I also never added this back when I switched to FTS for the fields in the db because it was too slow. The same happened with hashtags as well.

Eventually I did add caching of headings in gemtext bodies. So currently, AuraGem caches all gemtext headings, mimetypes, as well as the titles and publication dates if they occur in the file names or first level-1 headings, and other file metadata for audio files and a few other filetypes. The latest change I made was to keep track of link titles that link to resources on the same domain (internal links) so that if that file didn't have a title, it would use the link title. This change hasn't been used much in the index because I haven't indexed all of geminispace in about 2 months when this was added.

There's another thing that has been very volatile: backlinks. AuraGem Search's backlinks are actually quite recent. I believe I added them back in 2023, but they are extremely slow and sometimes it misses things depending on which order pages get crawled in. The biggest hamper on this is that searching for backlinks is way too slow currently.

AuraGem Search is actually closer to geminispace.info in that it takes a simpler approach by ignoring pagerank and other link ranking algorithms. It also never excluded links to filetypes that weren't gemtext and always included those in the index. I wrote quite a bit on why I chose not to use link ranking algorithms here:

I believe this has led to some really interesting search results, some improvements over TLGS and Kennedy, and some downsides. Much of this was investigated in this article:

I had hoped to make a second part to this that would do more investigation, and earlier this year I was going to write an update to this since AuraGem Search slightly changed since 2022. Unfortunately, I didn't have the time and my focus has shifted.

Regardless, AuraGem Search was a large failure for me, not because it didn't have good ideas, but because its progress was volatile. AuraGem Search is in a much better place now, and will now crawl Scroll and Nex (and plans for Spartan are coming soon), but I still consider it to be a failure.

The other failure is community acceptance. It is the only older Gemini Search Engine that was *never* listed on geminiprotocol.net (and formerly circumlunar.space's gemini docs; and it's also not on mozz.us), even though it's older than almost all of the other search engines that are listed, includng Kennedy and TLGS. It also is not used very much.

If I had to do things differently, I would have switched to FTS sooner, and I would have tried to make sure AuraGem/Ponix never went down. I would have also tried to get backlinks and the searching of other protocols in sooner.

Perhaps I'm too hard on it though. Since AuraGem Search predates Kennedy and TLGS, the lack of features in it early on are more understandable. Regardless, this capsule went down way too many times, and I think that is really the largest failure.

I do thank those that have used it, helped me test it, and promoted it. Even if it was a large failure, I still consider it to have been a good learning experience.

P.S. To those who want to look at the source code, here are the two repositories for the Search Engine and its crawler:

I have seen that Geminispace.info is going down. I never really cared for Geminispace.info, and I never used its predecessor, GUS, but I do have to say that it was the very inspiration that led me to create AuraGem Search, and so I thank GUS and Geminispace.info's developers for that.

2024-05-08 AuraGem Search Was an Utter Failure

A Farewell to Geminispace.info