Should we allow spiders in space

Last night I was searching for something on Google related to a post on someone else's gemlog. On accident autocorrect on my phone filled "gemini" into my full url, to which i clicked search as I was not paying enough attention. The first results found was my site (which is cross host in HTTP) pointing to the gemlog post I was researching. As the gmi to html proxies can work as a full on proxy, apparently Google had scrapped my page enough to index my response and the subsequent links it contained.

The part that was messed up is that the other person's post shows up under my domain.

Now this can easily be fixed through the issue of a robot.txt file, that isn't the issue. I think this brings up a bigger question of whether we should allow the web crawlers of the Complex Web dig through the content of the Simple Web.

Personally I don't mind the additional traffic. I host my capsule in both protocols so that if I write something that is relevant to a larger audience, there isn't a technology gap. While I really like the community here, I do also interact with other communities like the Fediverse. Just doesn't make sense to have multiple places for blogging if I don't have to.

Some part of me is worried about what more exposure can cause. We are seeing massive changes in culture over on Mastodon with the mass exodus of Twitter users. We saw it on Usenet back in the day. While those have an easier cost of entry than Gemini, it still feels weird to me having my capsule show up in Google. Gopher "died" because WWW started to take off, so there wasn't much abuse of the community. It still feels nice to be able to use gemini based search engines and have a small, correct set of results. Would be a shame if tons of garbage started piling up like it does on the web.

Or maybe I am worrying about nothing.

$ published: 2023-01-16 14:30 $

-- CC-BY-4.0 jecxjo 2023-01-16

back