-- Leo's gemini proxy

-- Connecting to ew.srht.site:1965...

-- Connected

-- Sending request

-- Meta line: 20 text/gemini


Re: Crawling

tags: smolnet

Alex Schröder wrote:

> What would a network look like that doesn't crawl?

> People say: that’s the only way a search engine can work. Maybe? Maybe not? What if sites sent updates, compiled databases? A bit like the Sitemap format? A sort of compiled and compressed word/URI index? And if then very few people actually sent in those indexes, would that not be a statement in itself? Now people don’t object because it takes effort. But perhaps they wouldn’t opt-in either!


Reading this struck some chords.

It is /not/ an unavoidable trait of this physical universe that there be crawlers --- possibly known as "nosy noisy hungry bots" :) --- Maybe they have to be noisy on the net, because they must somehow increase entropy in the universe :) I start to see the jungle --- yet I know, I would be killed on the very first contact in a real jungle.

I have contemplated the idea of buying some small virtual machine instance in a datacenter. But so far I have resisted, because I am not interested to guard this smol machine. It is a time sink on details I consider not interesting enough. And it seems to me, I'm not alone with this.

Thanks for writing this up!




-- Response ended

-- Page fetched on Wed Oct 20 20:35:03 2021