2024-03-06 Prepping AuraGem Search for Major Updates

I have been prepping AuraGem Search for some major updates recently. In order to expand the search engine into other protocols, I have added fields to the database and have started refactoring and rewritting parts of the crawler to be able to distinguish URL/protocol schemes.

For at least 2 years now my search engine has had a way of detecting which pages can be used as gemsub feeds and which cannot. So, I have also changed the crawler so that it can go into certain "modes" where it will only follow internal links (non-cross-host links). This will allow me to set a thread to periodically crawl feeds so that I can create an aggregator page. There are about 7822 gemsub feed pages in the index right now.

I have also started indexing links that request input from the user, including their prompts. These links are labelled "Input Prompt" in the search results. I will be adding a way to filter these out eventually, but for now they shouldn't really dominate the search results at all.

Lastly, I will be adding the ability to have the search engine almost instantly crawl your site by submitting your root url. This is similar to what some gopherspace search engines do, and I think it can be useful here on geminispace too.

AuraGem Search