-- Leo's gemini proxy
-- Connecting to bbs.geminispace.org:1965...
-- Connected
-- Sending request
-- Meta line: 20 text/gemini; charset=utf-8
Currently, there's the ability to limit displayed posts to recent ones, which is great.
On this page, there's a dynamic link to only display newer ones than the ones displayed on the recent posts page:
Unfortunately, this link is crawled by crawlers and ends up on search engine result pages. This is not ideal, because if I search for an uncommon keyword, where I can estimate the number of expected results, e.g. my last name, then I still get ~800 results, and the vast majority of them are links to Cosmos' "Show newer posts than this":
[deleted Geminispace.info search link for nordmeyer, which resulted in indexing the SERP because of a typo in geminispace.info/robots.txt]
Blocking `/~Cosmos/since` in robots.txt seems like a good idea.
Any thoughts on this?
2023-05-21 · 1 year ago
Agreed, I'll add that to robots.txt. That should alleviate the issue at least.
-- Response ended
-- Page fetched on Sun May 19 21:11:46 2024