2023-09-12 More Updates to AuraGem

Over the past two days I have made some more updates to AuraGem, particularly the Search Engine. These are mostly stability and QoL updates.

Search Engine:

Fragments are stripped in the crawler

The total indexed pages is now at over 340,000 pages and 2,178 domains.

Initial Backlinks support - you can now search a url and see a list of all backlinks to it

Pagination has been added to some of the "Indexed ..." pages. This vastly improved the stability of the gemini capsule.

Mimetypes pages have been limited to 300 results for now. In the future pagination will be added to these pages as well.

AuraGem Ask:

"Register Page" has been renamed to "Register Cert" so that it is more understandable

The "Register Cert" link shows up on more pages when a user has no client certificate active or when the current cert has not been registered with a username

Robots.txt:

The Sefaria proxy has been added to the robots.txt of pages that should not be crawled.