-- Leo's gemini proxy

-- Connecting to ew.srht.site:1965...

-- Connected

-- Sending request

-- Meta line: 20 text/gemini

2020-12-23

Re: Crawling

tags: smolnet


Alex Schröder wrote:


> What would a network look like that doesn't crawl?


> People say: that’s the only way a search engine can work. Maybe? Maybe not? What if sites sent updates, compiled databases? A bit like the Sitemap format? A sort of compiled and compressed word/URI index? And if then very few people actually sent in those indexes, would that not be a statement in itself? Now people don’t object because it takes effort. But perhaps they wouldn’t opt-in either!


gemini://alexschroeder.ch/page/2020-12-22_Crawling


Reading this struck some chords.


It is /not/ an unavoidable trait of this physical universe that there be crawlers --- possibly known as "nosy noisy hungry bots" :) --- Maybe they have to be noisy on the net, because they must somehow increase entropy in the universe :) I start to see the jungle --- yet I know, I would be killed on the very first contact in a real jungle.


I have contemplated the idea of buying some small virtual machine instance in a datacenter. But so far I have resisted, because I am not interested to guard this smol machine. It is a time sink on details I consider not interesting enough. And it seems to me, I'm not alone with this.


Thanks for writing this up!


Cheers,

~ew



Home

-- Response ended

-- Page fetched on Wed Oct 20 20:35:03 2021