2020-12-23

Re: Crawling

tags: smolnet

Alex Schröder wrote:

> What would a network look like that doesn't crawl?

> People say: that’s the only way a search engine can work. Maybe? Maybe not? What if sites sent updates, compiled databases? A bit like the Sitemap format? A sort of compiled and compressed word/URI index? And if then very few people actually sent in those indexes, would that not be a statement in itself? Now people don’t object because it takes effort. But perhaps they wouldn’t opt-in either!

gemini://alexschroeder.ch/page/2020-12-22_Crawling

local copy

Reading this struck some chords.

It is /not/ an unavoidable trait of this physical universe that there be crawlers --- possibly known as "nosy noisy hungry bots" :) --- Maybe they have to be noisy on the net, because they must somehow increase entropy in the universe :) I start to see the jungle --- yet I know, I would be killed on the very first contact in a real jungle.

I have contemplated the idea of buying some small virtual machine instance in a datacenter. But so far I have resisted, because I am not interested to guard this smol machine. It is a time sink on details I consider not interesting enough. And it seems to me, I'm not alone with this.

Thanks for writing this up!

Cheers,

~ew

Home