-- Leo's gemini proxy
-- Connecting to gemini.techrights.org:1965...
-- Connected
-- Sending request
-- Meta line: 20 text/gemini;lang=en-GB
00:13 *u-amarsh04 has quit (Connection closed)
00:13 *u-amarsh04 has quit (connection closed)
06:33 *DaemonFC has quit (Quit: Leaving)
07:43 schestowitz-TR; restarted gewmini proxy
07:43 schestowitz-TR; lately it has been having issues after 2 months of smooth sailing
07:43 schestowitz-TR; this predates any code changes
08:26 Techrights-sec; any log records regardingthe freezes? systemd should have restarted it if
08:26 Techrights-sec; it had stopped running and exited
08:26 Techrights-sec; /var/log/nginx/*
08:27 schestowitz-TR; it seems to sort of hang rather than crash. It has generally been running mow slowly (longer load times) recently, but didn't completely fail until about a week ago. I did not apply any system updates or anything...
08:39 schestowitz; "
08:39 schestowitz; 2022/04/16 07:40:22 [error] 29508#29508: *11165 connect() to unix:/var/run/nginx/nginx.gemini.proxy.0.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 83.202.186.108, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2020/05/27/canonical-helps-microsoft/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org", referrer: "http://
08:39 schestowitz; techrights.org/"
08:40 Techrights-sec; hmm. it will be hard to figure out what makes it hang without specific
08:40 Techrights-sec; log entries from /var/log/nginx/ and maybe even not then
08:40 Techrights-sec; my guess though is that the http response from TR is tripping it up
08:40 Techrights-sec; since that is not reliable and most recent software including python
08:40 Techrights-sec; expet the network to both be 100% availabe and 100% perfectly reliable
08:40 schestowitz-TR; checking...
08:43 schestowitz-TR; of course one lusy workaround is to restart the service every hour
08:43 schestowitz-TR; one thing I nopticed is that it slowed down a lot
08:43 schestowitz-TR; also, to restart it that may take like 2 minutes
08:43 schestowitz-TR; which seems far too long
08:43 schestowitz-TR; I saw no indication of memory going low
08:49 Techrights-sec; based on the shell script problems, python is probably waiting for
08:49 Techrights-sec; a dropped HTTP transaction which will never finish to finish; maybe
08:49 Techrights-sec; there is a way to put in a timeout
08:49 Techrights-sec; scratch that, the question would be Gemini protocol and Perl not HTTP and python
08:51 schestowitz-TR; what intrigued me is that it ran OK for 2 months, no reboots or updates, and then suddenly the problems started
08:51 schestowitz-TR; afaik, we did not change anythiing
08:51 schestowitz-TR; except maybe adding the tc stuff and changing sudoers
08:51 schestowitz-TR; maybe when tc gets run and there is an open nginx connection it gets it worked out
08:51 schestowitz-TR; like stopping a packet?
08:56 Techrights-sec; try git now
08:56 schestowitz-TR; manually saw changelog.changes, applying now... will restart
08:59 schestowitz-TR; done
08:59 schestowitz-TR; I am going to tail -f error.log
08:59 schestowitz-TR; as times there correspond with those cases of proxy downtimes
08:59 schestowitz-TR; for now it is empty
08:59 schestowitz-TR; btw, did you see the LF article?
09:02 schestowitz-TR; it is the very latest post in TR
09:02 schestowitz-TR; it kept me up until 2am, way beyond my sleep time, as I wanted the upload to finish and be verified with md5
09:02 schestowitz-TR; I think the timing of the report (easter friday) says how LF itself feels about it
09:02 schestowitz-TR; this has already been discussed a lot in IRC, where I floated a first draft of it
09:05 Techrights-sec;
09:05 Techrights-sec; tahnks
09:05 Techrights-sec; ^thanks
09:05 Techrights-sec; I'll need to refresh my memory on the LF article, can you send the URL again?
09:05 Techrights-sec; THe one about the CoC? Yes, I read it. It looks likely that
09:05 Techrights-sec; LF is using it to silence anyone not toeing the line, especially in regards
09:05 Techrights-sec; to M$ insurgency
09:05 Techrights-sec; Yes, releasing on not just late on Friday but a very major holiday for most
09:05 Techrights-sec; of the world is a strong indication they wish to bury such awareness
09:05 Techrights-sec; Not that there are any journalists left to cover it, there are so few and
09:05 Techrights-sec; those few are already so busy
09:05 schestowitz-TR; remember that now to criticise criminals is the moral equivalent of kicking little innocent puppies
09:05 Techrights-sec; yes they are pushing that narrative hard nowadays
09:05 Techrights-sec; It's "unprofessional".
09:09 schestowitz-TR; we agree 100% on this
09:09 schestowitz-TR; no point debating this further
09:09 schestowitz-TR; I don't suppose any "news" site will pick up the topic
09:09 schestowitz-TR; that's x-ist
09:09 schestowitz-TR; "it helps the [some bad group]"
09:13 schestowitz-TR; there sre sying along the lines of
09:13 schestowitz-TR; "if you don't offer legal options, then ./.. piracy"
09:13 schestowitz-TR; "if you suppress subject x... then [some consequences"
09:13 schestowitz-TR; BN and TR did well over the years because they do offer an outlet for suppressed views
09:13 schestowitz-TR; but the perpetual risk is being taken out of context and presented as "Extremists"
09:13 schestowitz-TR; as happend to FSF, Pocock and others
09:13 schestowitz-TR; it's a "delicate dance"
09:13 schestowitz-TR; all in all I think we did alright
09:13 schestowitz-TR; it's very hard to argue that we're extreme
09:13 schestowitz-TR; gulagtroll is still trying
09:13 schestowitz-TR; like provoking DFC with certain subjects, then linking to that in Musk Social
09:17 Techrights-sec; ack
09:17 Techrights-sec; yes he is still trolling
09:24 schestowitz-TR; btw, irc is suitable for discussions like sheela zemlin
09:24 schestowitz-TR; because that is very informal
09:24 schestowitz-TR; hardly counts as "stalking" or "harassment"
09:24 schestowitz-TR; the subject is VERY relevant
09:24 schestowitz-TR; but you would not want to talk about it much in articles
09:24 schestowitz-TR; nor will you wish to mention anything race--related
09:24 schestowitz-TR; or cultural differences
09:24 schestowitz-TR; it might be OK foirm people from rich nations
09:24 schestowitz-TR; I still think many CEO appoinments in the US
09:24 schestowitz-TR; are based not on skills
09:24 schestowitz-TR; but based on a baggage associated with public opinion
09:24 schestowitz-TR; seeing that some of them (accent aside) cannot speak coherently
09:25 Techrights-sec; :(
09:25 Techrights-sec; yes if even that. same for nearly all other CxO appointements
09:25 Techrights-sec; The C-suite as it is called just seems to shuffle between failures at
09:25 Techrights-sec; various businesses
09:25 schestowitz-TR; usually they do not do actual work
09:25 schestowitz-TR; they just pass 'email' (Outlook) and tell peopple to do things
09:29 Techrights-sec; Outlook/Exchange is used for plausible deniability
09:29 Techrights-sec; never a real mail service
09:29 schestowitz-TR; microsoft <3 outlook
09:29 schestowitz-TR; russia <3 outlook
09:29 schestowitz-TR; nsa <3 outlook
09:29 schestowitz-TR; stupid MBAs <3 stupid magazines that tell them criminals from microsoft = "professionalism"
09:37 Techrights-sec; yep
09:37 Techrights-sec; probably the main reason that putin can't attack the computers is because
09:37 Techrights-sec; russia itself is open to a devastating counter attack due to its failure
09:37 Techrights-sec; 4x to eliminate M$ products from within its borders. At least some of the
09:37 Techrights-sec; time that was due to out of control corruption which then resulted in firing
09:37 Techrights-sec; of many who knew computers and relaced them with microsofters
10:07 schestowitz-TR; at around 9:01am the error logs started filling up again
10:07 schestowitz-TR; I've just restarted the service, about an hour late
10:08 schestowitz; 2022/04/16 10:05:51 [error] 29508#29508: *11484 connect() to unix:/var/run/nginx/nginx.gemini.proxy.0.sock failed (111: Connection refused) while connecting to upstream, client: 201.65.94.22, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2021/04/04/microsoft-nationalism/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org", referrer: "http://techrights.org/"
10:08 schestowitz; 2022/04/16 10:06:49 [error] 29508#29508: *11488 connect() to unix:/var/run/nginx/nginx.gemini.proxy.0.sock failed (111: Connection refused) while connecting to upstream, client: 81.154.169.249, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/git/tr-git/IPFS/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org"
10:08 -TechrightsBN/#boycottnovell-techrights.org | Welcome to Techrights
10:12 Techrights-sec; ack
10:12 Techrights-sec; thanks
10:16 *u-amarsh04 (~amarsh04@joseon-rmogvn.g0d7.dtdf.mc4289.IP) has joined #boycottnovell
10:16 *u-amarsh04 (~amarsh04@v6xmmrhxmbafc.irc) has joined #boycottnovell
10:29 schestowitz; -rw-r----- 1 www-data adm 114391 Apr 16 10:28 error.log
10:29 schestowitz; -rw-r----- 1 www-data adm 59570 Apr 15 23:57 error.log.1
10:29 schestowitz; -rw-r----- 1 www-data adm 2756 Apr 6 23:53 error.log.10.gz
10:29 schestowitz; -rw-r----- 1 www-data adm 2141 Apr 5 23:35 error.log.11.gz
10:29 schestowitz; -rw-r----- 1 www-data adm 2764 Apr 4 23:27 error.log.12.gz
10:29 schestowitz; -rw-r----- 1 www-data adm 3249 Apr 3 23:52 error.log.13.gz
10:29 schestowitz; -rw-r----- 1 www-data adm 4320 Apr 2 23:29 error.log.14.gz
10:29 schestowitz; -rw-r----- 1 www-data adm 3319 Apr 14 23:58 error.log.2.gz
10:29 schestowitz; -rw-r----- 1 www-data adm 7667 Apr 13 23:45 error.log.3.gz
10:29 schestowitz; -rw-r----- 1 www-data adm 23349 Apr 12 23:25 error.log.4.gz
10:29 schestowitz; -rw-r----- 1 www-data adm 2112 Apr 11 23:15 error.log.5.gz
10:29 schestowitz; -rw-r----- 1 www-data adm 1885 Apr 10 22:33 error.log.6.gz
10:29 schestowitz; -rw-r----- 1 www-data adm 1613 Apr 9 23:14 error.log.7.gz
10:29 schestowitz; -rw-r----- 1 www-data adm 3112 Apr 8 23:21 error.log.8.gz
10:29 schestowitz; -rw-r----- 1 www-data adm 2657 Apr 7 23:28 error.log.9.gz
10:29 schestowitz-TR; now it barely lasts a minute eveven if or when I restart the services
10:29 schestowitz-TR; looking into it
10:34 schestowitz; systemctl status gemini-proxy.service
10:34 schestowitz; gemini-proxy.service - Gemini Proxy Service
10:34 schestowitz; Loaded: loaded (/etc/systemd/system/gemini-proxy.service; enabled; vendor preset: enabled)
10:34 schestowitz; Active: inactive (dead) since Sat 2022-04-16 10:33:28 BST; 1min 3s ago
10:38 schestowitz-TR; it looks like with the latest git change is became more sensitive
10:38 schestowitz-TR; it also looks like systemd does try to restart it a few times
10:59 Techrights-sec; ok some more changes in the works
10:59 Techrights-sec; ok sevral changes in Git now
10:59 schestowitz-TR; running for 2 mins now, no errors yet
11:03 schestowitz-TR; so far so good
11:03 schestowitz; 2022/04/16 11:03:17 [error] 11909#11909: *58 FastCGI sent in stderr: "[Sat Apr 16 11:03:17 2022] gemini-proxy.pl: Failed to connect: Resource temporarily unavailable, 'SSL wants a read first'" while reading response header from upstream, client: 93.218.25.16, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2022/04/14/spurning-science/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "
11:03 schestowitz; gemini.techrights.org", referrer: "http://techrights.org/
11:03 -TechrightsBN/#boycottnovell-techrights.org | Welcome to Techrights
11:12 Techrights-sec; ok
11:12 Techrights-sec; checking
11:12 Techrights-sec; (fwiw it works here in the local test set up)
11:12 Techrights-sec; still checking
11:12 Techrights-sec; strange only the path is different
11:12 Techrights-sec; that is the matter,
11:12 Techrights-sec; where would be a good place to cache a copy of the public cert for
11:12 Techrights-sec; gemini.techrihgts.org ? Somewhere in /etc?
11:13 schestowitz-TR; maybe userspace?
11:14 schestowitz; 2022/04/16 11:07:55 [error] 11909#11909: *75 connect() to unix:/var/run/nginx/nginx.gemini.proxy.0.sock failed (111: Connection refused) while connecting to upstream, client: 93.23.18.203, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2021/11/10/firefox-esr-91-issues/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org", referrer: "http://techrights.org/"
11:14 schestowitz; 2022/04/16 11:08:27 [error] 11909#11909: *77 connect() to unix:/var/run/nginx/nginx.gemini.proxy.0.sock failed (111: Connection refused) while connecting to upstream, client: 84.249.218.214, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2020/08/31/linux-should-reject-github/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org", referrer: "http://techrights.org/"
11:14 schestowitz; 2022/04/16 11:09:07 [crit] 11909#11909: *89 SSL_do_handshake() failed (SSL: error:141CF06C:SSL routines:tls_parse_ctos_key_share:bad key share) while SSL handshaking, client: 51.159.99.253, server: 0.0.0.0:443
11:14 schestowitz; 2022/04/16 11:09:35 [error] 11909#11909: *90 connect() to unix:/var/run/nginx/nginx.gemini.proxy.0.sock failed (111: Connection refused) while connecting to upstream, client: 95.115.120.75, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2019/04/19/pypy-7-1-1/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org", referrer: "http://techrights.org/"
11:14 schestowitz; 2022/04/16 11:10:44 [error] 11909#11909: *93 connect() to unix:/var/run/nginx/nginx.gemini.proxy.0.sock failed (111: Connection refused) while connecting to upstream, client: 84.249.218.214, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2020/06/18/rusty-censorship/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org", referrer: "http://techrights.org/"
11:15 Techrights-sec; /home/gemini/bin/Gemini-Proxy/Cert-Cache/gemini.tecrights.org.pem ?
11:17 Techrights-sec; $ git diff
11:17 Techrights-sec; diff --git a/Gemini/Gemini-Proxy/gemini-proxy.pl b/Gemini/Gemini-Proxy/gemini-proxy.pl
11:17 Techrights-sec; index d861b5e..5c8b4ce 100755
11:17 Techrights-sec; --- a/Gemini/Gemini-Proxy/gemini-proxy.pl
11:17 Techrights-sec; +++ b/Gemini/Gemini-Proxy/gemini-proxy.pl
11:17 Techrights-sec; @@ -108,10 +108,10 @@ sub fetch_gemini {
11:17 Techrights-sec; Timeout=>2,
11:17 Techrights-sec; Proto => 'tcp',
11:17 Techrights-sec; # SSL_ca_path => '/home/pi/Certs/',
11:17 Techrights-sec; - SSL_ca_file => '/home/gemini/bin/Gemini-Proxy/Cert-Cache/gemini.tecrights.org.pem',
11:18 Techrights-sec; + # SSL_ca_file => '/home/gemini/bin/Gemini-Proxy/Cert-Cache/gemini.tecrights.org.pem',
11:18 Techrights-sec; SSL_hostname => "gemini.techrights.org",
11:18 Techrights-sec; - SSL_verify_mode => SSL_VERIFY_PEER, # overriden by next line ...
11:18 Techrights-sec; - # SSL_verify_mode => SSL_VERIFY_NONE, # unsafe kludge
11:18 Techrights-sec; + # SSL_verify_mode => SSL_VERIFY_PEER, # overriden by next line ...
11:18 Techrights-sec; + SSL_verify_mode => SSL_VERIFY_NONE, # unsafe kludge
11:18 Techrights-sec; )
11:18 Techrights-sec; or die("Failed to connect: $!, '$SSL_ERROR'\n");
11:20 Techrights-sec; maybe the NAT hairpinning (NAT loopback) is causing trouble with the
11:20 Techrights-sec; host IP + name + SSL combination?
11:20 Techrights-sec; Anyway, try the above diff
11:20 schestowitz-TR; it does not appear to be in git, doy ou want me to apply the change manually?
11:27 Techrights-sec;
11:27 Techrights-sec; if it's possible
11:27 Techrights-sec; thanks
11:27 schestowitz-TR; changing, will restart
11:27 schestowitz-TR; 111 access forbidden by rule...
11:28 schestowitz; 2022/04/16 11:23:42 [error] 11909#11909: *111 access forbidden by rule, client: 23.100.232.233, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2012/05/12/antisocial-element-at-microsoft/ HTTP/1.1", host: "gemini.techrights.org", referrer: "http://techrights.org/2012/05/12/antisocial-element-at-microsoft/"
11:28 schestowitz; 2022/04/16 11:24:06 [error] 11909#11909: *112 access forbidden by rule, client: 52.162.211.179, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2018/07/08/striving-to-improve-patent-quality/ HTTP/1.1", host: "gemini.techrights.org", referrer: "http://techrights.org/2018/07/08/striving-to-improve-patent-quality/"
11:28 -TechrightsBN/#boycottnovell-techrights.org | Charles Manson, the Unabomber, and Microsoft | Techrights
11:28 schestowitz; 2022/04/16 11:24:52 [error] 11909#11909: *113 FastCGI sent in stderr: "[Sat Apr 16 11:24:52 2022] gemini-proxy.pl: Failed to connect: Resource temporarily unavailable, 'SSL wants a read first'" while reading response header from upstream, client: 81.154.169.249, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/git/files/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.
11:28 -TechrightsBN/#boycottnovell-techrights.org | In Spite of Resistance From the Patent Microcosm the USPTO Strives to Improve Patent Quality | Techrights
11:28 schestowitz; org", referrer: "http://gemini.techrights.org/proxy?url=gemini://gemini.techrights.org/git/"
11:28 schestowitz; 2022/04/16 11:26:47 [error] 11909#11909: *117 FastCGI sent in stderr: "[Sat Apr 16 11:26:47 2022] gemini-proxy.pl: Failed to connect: Resource temporarily unavailable, 'SSL wants a read first'" while reading response header from upstream, client: 84.146.40.202, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2021/03/15/duckduckgo-in-2021/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "
11:28 schestowitz; gemini.techrights.org", referrer: "http://techrights.org/"
11:28 -TechrightsBN/#boycottnovell-gemini.techrights.org | Gemini Proxy for Techrights
11:28 -TechrightsBN/#boycottnovell-techrights.org | Welcome to Techrights
11:30 *DaemonFC (~daemonfc@k5ghkuun6w7re.irc) has joined #boycottnovell
11:38 Techrights-sec; is that with the diff?
11:40 schestowitz; 2022/04/16 11:39:50 [warn] 19740#19740: "ssl_stapling" ignored, issuer certificate not found for certificate "/etc/ssl/certs/nginx-selfsigned.crt"
11:40 schestowitz; 2022/04/16 11:39:50 [warn] 19741#19741: "ssl_stapling" ignored, issuer certificate not found for certificate "/etc/ssl/certs/nginx-selfsigned.crt"
11:40 schestowitz; 2022/04/16 11:40:15 [error] 19743#19743: *1 access forbidden by rule, client: 23.100.232.233, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/2010/09/16/microsoft-latin-america-vs-freedom-sw/ HTTP/1.1", host: "gemini.techrights.org", referrer: "http://techrights.org/2010/09/16/microsoft-latin-america-vs-freedom-sw/"
11:40 -TechrightsBN/#boycottnovell-techrights.org | Microsoft Pulls a Russia in Brazil | Techrights
11:40 schestowitz-TR; yes, but let me check again
11:40 schestowitz-TR; ok, seems like I edited this correctly
11:40 schestowitz-TR; btu I've just restarted nginx as well
11:52 Techrights-sec; try /tmp/gemini-proxy.pl that should be the old version
11:53 schestowitz; 2022/04/16 11:52:38 [error] 20752#20752: *5 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 81.154.169.249, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/git/sloc.gmi HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org", referrer: "http://gemini.techrights.org/proxy?url=gemini://gemini.techrights.org/git/"
11:53 schestowitz; 2022/04/16 11:52:56 [error] 20752#20752: *1 FastCGI sent in stderr: "[Sat Apr 16 11:52:56 2022] gemini-proxy.pl: Failed to connect: Resource temporarily unavailable, 'SSL wants a read first'" while reading response header from upstream, client: 80.220.101.127, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org"
11:53 schestowitz; 2022/04/16 11:53:01 [error] 20752#20752: *5 FastCGI sent in stderr: "[Sat Apr 16 11:53:01 2022] gemini-proxy.pl: Failed to connect: Resource temporarily unavailable, 'SSL wants a read first'" while reading response header from upstream, client: 81.154.169.249, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/git/sloc.gmi HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.
11:53 schestowitz; org", referrer: "http://gemini.techrights.org/proxy?url=gemini://gemini.techrights.org/git/"
11:53 schestowitz; 2022/04/16 11:53:10 [error] 20752#20752: *9 FastCGI sent in stderr: "[Sat Apr 16 11:53:10 2022] gemini-proxy.pl: Failed to connect: Resource temporarily unavailable, 'SSL wants a read first'" while reading response header from upstream, client: 80.220.101.127, server: _, request: "GET /proxy?url=gemini://gemini.techrights.org/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/nginx/nginx.gemini.proxy.0.sock:", host: "gemini.techrights.org"
12:19 Techrights-sec; seems to work in place now
12:19 Techrights-sec; hmm now it shows "502 Bad Gateway"
12:19 Techrights-sec; over at gemini.techrights.org
12:19 Techrights-sec; it appears to not be running, according to ps and pgrep
12:19 schestowitz-TR; it died after a while
12:25 Techrights-sec; I am not sure at all what is killing it
12:25 schestowitz-TR; nothing has changed on the server aroudund then except sudoers and tc in scripts
12:29 schestowitz-TR; it also became a lot slower to respond
12:30 schestowitz-TR; it used to be very snappy
12:30 schestowitz-TR; now, even when it works, it can take several seconds
12:30 schestowitz-TR; lxer has just posted some github crap
12:30 schestowitz-TR; they don't like their readers
12:33 *Noisytoot has quit (Ping timeout: 2m30s)
12:41 Techrights-sec; I presume something has changed in the environment?
12:44 Techrights-sec; techrights.org
12:44 Techrights-sec; gemini.techrights.org is an alias for home.techrights.org.
12:44 Techrights-sec; home.techrights.org has address 81.154.169.249
12:44 Techrights-sec; real 0m1.503s
12:44 Techrights-sec; user 0m0.027s
12:44 Techrights-sec; sys 0m0.027s
12:44 Techrights-sec; 1.5 seconds :(
12:44 Techrights-sec; thinking
12:44 Techrights-sec; $ time host gemini.techrights.org
12:44 Techrights-sec; gemini.techrights.org is an alias for home.techrights.org.
12:44 Techrights-sec; home.techrights.org has address 81.154.169.249
12:44 Techrights-sec; real 0m1.503s
12:44 Techrights-sec; user 0m0.027s
12:44 Techrights-sec; sys 0m0.027s
12:44 schestowitz-TR; afaik, i could only see issues after 2 months of smooth sailing around the time we added c to the video update script and
12:44 schestowitz-TR; the capsule update script
12:45 schestowitz-TR; some days later I also killed and restarted ipfs
12:45 schestowitz-TR; because of the connection issues
12:45 schestowitz-TR; which left me using the wifi on another channel
12:45 schestowitz-TR; but that was days after
12:45 schestowitz-TR; I think that was days after the proxy isues
12:45 Techrights-sec; thinking
12:45 Techrights-sec; It's an intermittent problem
12:59 schestowitz; gemini pts/8 tmux(5296).%12 Sat Apr 16 10:08 gone - no logout
12:59 schestowitz-TR; is this you?
13:21 Techrights-sec; probably
13:21 Techrights-sec; I have gemini with tmux atm
13:23 schestowitz-TR; I think that around the 10th was the first time I needed to restart ngnix but then I realised
13:23 schestowitz-TR; the proxy server needed to be restarted after 1 montk, 30 days
13:23 schestowitz-TR; sometimes it takes a couple of minutes just to restart
13:23 schestowitz-TR; but pages were then slow to load
13:23 schestowitz-TR; and it would not be the last restart needed, I think another was needed later in the same day
13:27 schestowitz-TR; evewn if I got it to self-heal
13:27 schestowitz-TR; that would still not explain why it got so much slower
13:27 schestowitz-TR; and I suspect that the slowness is the cause of the crashes or is related to it
13:27 schestowitz-TR; the capsule, gemini://, is super fact from here
13:27 schestowitz-TR; the proxy is not
13:30 Techrights-sec; ack
13:30 Techrights-sec; the capsule does not use DNS though, the proxy does at least one lookup per
13:30 Techrights-sec; request
13:30 Techrights-sec; the requests seem to take well oveer 1 second for DNS
13:30 Techrights-sec; so that might be part of the problem or just one symptom
13:30 Techrights-sec; $ time host gemini.techrights.org
13:30 Techrights-sec; gemini.techrights.org is an alias for home.techrights.org.
13:30 Techrights-sec; home.techrights.org has address 81.154.169.249
13:30 Techrights-sec; real 0m5.109s
13:30 Techrights-sec; user 0m0.021s
13:30 Techrights-sec; sys 0m0.051s
13:30 *DaemonFC has quit (Quit: Leaving)
13:36 Techrights-sec; that's from your RPi
13:36 Techrights-sec; [
13:36 Techrights-sec; perhaps, if they overlap just right; yes it varies a lot but is always slow
13:37 schestowitz-TR; if I run this from my machine or root at pi it's a lot faster\
13:37 schestowitz-TR; oh, actually it varies from the pi
13:37 schestowitz-TR; even over a second sometimes
13:37 schestowitz-TR; OK, so I think it narrows down the issue
13:37 schestowitz-TR; maybe a TC thing
13:37 schestowitz-TR; is it possible that at one point two scripts called the tc script at the same time, resulting in a bad state?
13:37 schestowitz-TR; i have just run tc-0shaper-06
13:37 schestowitz-TR; and the response is faster
13:37 schestowitz-TR; wait, it is no faster, it varies a lot
13:37 schestowitz-TR; even over 6 secs at one point
13:37 schestowitz-TR; so I think this is def. the source of the problem
13:40 Techrights-sec; likely
13:42 schestowitz-TR; video updates run tc swapper
13:42 schestowitz-TR; they start every three hours past 20
13:42 schestowitz-TR; planet every 3 hours but no tc there
13:42 schestowitz-TR; it looks like you commented out the global rrss updater
13:43 schestowitz-TR; so it runs every hour, past 6
13:43 schestowitz-TR; the commented out bit says every 3 hours
13:43 schestowitz-TR; was this change recent?
13:45 schestowitz-TR; crontab -l | grep updater
13:45 schestowitz-TR; this would reduce reliability as it increases the chance of a stale lock file
13:45 schestowitz-TR; sometimes I run a ls comment to check the time of the latest lock
13:45 schestowitz-TR; and manually remove it if it's old
13:45 Techrights-sec; not that recent. I mentioned it but did not update the text as I was waiting
13:45 Techrights-sec; to see if it caused an increase in reliable updates. Too often the updates
13:45 Techrights-sec; hang and then fail.
13:45 Techrights-sec; That's another problem to track down, the scripts are supposed to remove the
13:45 Techrights-sec; their lockfiles upon exit regardless of the reason for exiting
13:48 schestowitz-TR; i think what we have here is another 'blessing in disguise' (like with obs after noisetorch issue)
13:48 schestowitz-TR; as the proxy issue is def. connected to lookup latecy, which it can eventualyl choke on
13:48 schestowitz-TR; and to tackle this issue we can also ackle the lockfile issue and tc
13:48 schestowitz-TR; overlapping scripts can contribute to the issue, I think
13:48 Techrights-sec; I've already relabled the lockfiles so that each script uses a unique name
13:50 schestowitz-TR; one thing i've noticed is, sometimes the scripts do not terminate
13:50 schestowitz-TR; they leave ffprobe behind
13:50 schestowitz-TR; and also leave themselves behind
13:50 schestowitz-TR; an hour ago I killed manually several "updaters"
13:50 schestowitz-TR; they might be the reason lock files are left behind
13:50 schestowitz-TR; so in a sense our pileup of scripts with system calls maybe made its way to the proxy
13:50 schestowitz-TR; and the proxy issue is just a symptom
13:50 schestowitz-TR; hence not predictable
13:50 schestowitz-TR; but I suppose more robust after some code changes
13:50 Techrights-sec; yes, I was just thinking of that too
13:53 schestowitz-TR; Yes, I got the same, but it's OK now
13:53 schestowitz-TR; I was about to ask if you ran some command
13:53 Techrights-sec; DNS is not supposed to take 6 seconds
13:53 Techrights-sec; Failed to connect to the server: dial tcp 81.154. 169.249:1965: i/o timeout.
13:53 Techrights-sec; ^gemini
13:57 Techrights-sec; I tried lsof but don't have any use of it
14:00 schestowitz-TR; maybe reset tc if such a concept exists?
14:00 schestowitz-TR; until we get saner dns again?
14:00 Techrights-sec; I tried lsof but don't have any use of it
14:00 Techrights-sec; the tc scripts should clear the queues
14:09 schestowitz-TR; is it you running tcpdump?
14:09 schestowitz-TR; process 15593
14:09 schestowitz-TR; the pi is clocking more than 1MB/sec sometimes
14:09 schestowitz-TR; so I think that might choke all traffic inc. DNS
14:09 schestowitz-TR; not sure, let me shrottle it more
14:09 Techrights-sec; IPFS?
14:18 schestowitz-TR; shutting down ipfs to test
14:18 schestowitz-TR; now the proxy seems snappier
14:18 schestowitz-TR; I wonder if ipfs itself became more b/w-heavy
14:18 schestowitz-TR; I did not notice anything different on other machines
14:18 schestowitz-TR; restarting ipfs
14:18 schestowitz-TR; its traffic level are saner, for now
14:18 schestowitz-TR; and the proxy is responsive and fast
14:18 Techrights-sec; ack
14:20 schestowitz-TR; i suppose the fuller effect will be known when scripts run with tc in them but I think we at least have a far better
14:20 schestowitz-TR; understanding now of what goes on
14:21 schestowitz-TR; according to my router the pi down about 10gb a day up+down
14:21 Techrights-sec; which ports is that distributed oveer?
14:21 Techrights-sec; Is most IPFS?
14:22 schestowitz-TR; for sure, but there is no breakdown by port numbers etc.
14:22 schestowitz-TR; I'm going to monitor the services and things for the next few hours
14:22 schestowitz-TR; it seems like the proxy is at least fast again and does not fall over, for now...
14:22 schestowitz-TR; afaik, in technical terms we're doing many things other capsules do not do
14:22 schestowitz-TR; so we are in the area of "undocumented"
14:24 *Noisytoot (~noisytoot@y56qsb6r6yg94.irc) has joined #boycottnovell
14:25 Techrights-sec; ack
17:15 *u-amarsh04 has quit (Quit: Konversation terminated!)
17:15 *u-amarsh04 has quit (Quit: Konversation terminated!)
17:20 schestowitz-TR; https://nitter.eu/FOSSForce/status/1515352822946086914#m
17:20 -TechrightsBN/#boycottnovell-nitter.eu | FOSS Force (@FOSSForce): "GNU Core Utilities: coreutils-9.1 released | Tux Machines https://buff.ly/3Og74pB" | nitter
17:21 schestowitz-TR; https://nitter.eu/geeknik/status/1515345773856182278#m
17:21 -TechrightsBN/#boycottnovell-nitter.eu | Geeknik`s {}; Farm (@geeknik): "Whats New in Fedora 36 http://www.tuxmachines.org/node/163768" | nitter
17:21 schestowitz-TR; https://nitter.eu/BrideOfLinux/status/1515329239632986119#m
17:21 -TechrightsBN/#boycottnovell-nitter.eu | Christine Hall (@BrideOfLinux): "Still kicking after all these years: GNU Core Utilities: coreutils-9.1 released | Tux Machines https://buff.ly/3Og74pB" | nitter
18:25 schestowitz-TR; backing upo pi:/home to external usb at the moment
18:25 schestowitz-TR; the usb backup holds the whole system inclluding OS
18:25 schestowitz-TR; we have not made many changes at that level
18:25 schestowitz-TR; regarding the proxy, I need to manually restart it every now and then every 30 minutes or so
18:25 schestowitz-TR; so we did not fix the issue, but we better understand what the issue is
18:25 schestowitz-TR; and when it runs, the lag is at least minimal
18:25 schestowitz-TR; as before
18:25 schestowitz-TR; I've now cleaned ALL my RSS feeds except the patent stuff!
18:25 schestowitz-TR; backup during the holiday is a priority
18:25 Techrights-sec; ack
18:25 Techrights-sec; maybe a cron job. I suppose it should be rewritten to have several parallel
18:25 Techrights-sec; processes going.
18:25 Techrights-sec; So that if one slows down or has to wait, other connections can still happen.
18:25 schestowitz-TR; yes, before cronn ing everything I just want to better understand the uptime
18:25 schestowitz-TR; it did run perfectly OK for 2 months
18:32 schestowitz-TR; the pi is immensely valuable to me
18:32 schestowitz-TR; with it, my "working notes" get replicated to 3 places
18:32 schestowitz-TR; external usb, pi, and desktop over ssh with cache
18:32 schestowitz-TR; it's now in another floor, upstairs
18:32 schestowitz-TR; if ipfs goes down and I'm afak, I will know
18:32 schestowitz-TR; blinking
18:32 schestowitz-TR; over gemini it has served about 4 million pages
18:32 schestowitz-TR; and over ipfs TBs of data
18:32 schestowitz-TR; rianne uses hers for work, with a 28-inch screen and lots of monitoring, 2 or 3 VPNs
18:32 schestowitz-TR; it is used even when I'm on shift (rpi400)
18:32 Techrights-sec; excellent!
18:32 Techrights-sec; ack
18:32 Techrights-sec; I suppose the soldering will never be convenient to try
18:32 Techrights-sec; yay!
18:33 schestowitz-TR; at the moment I cannot think of a use for buttons on the device as even a reboot button would
18:33 schestowitz-TR; never get used
18:33 schestowitz-TR; so from a practical POV, it's a case of "wait, I'll find use for it later on... maybe another
18:33 schestowitz-TR; device"
18:34 Techrights-sec; She has a model 400?
18:34 Techrights-sec; yes there are other "HAT"s which can do the button thing without needing
18:34 Techrights-sec; to be soldered
18:35 schestowitz-TR; so far since the morning no lock fiels got 'jammed'
18:35 schestowitz-TR; i think at one point I'll run some script to 'clean up' orphaned and incomplete runs
18:35 schestowitz-TR; ffmpeg probes etc.
18:36 Techrights-sec; great
18:38 schestowitz-TR; recently I noticed we get about 20-30% more unique IPs in gemini then lastt year
18:38 schestowitz-TR; takern on a daily basis
18:38 schestowitz-TR; drew is taking a break for gemini to spend more time smokign weed :-)
18:38 schestowitz-TR; *from gemini
18:38 schestowitz-TR; That's fine, he made some good tools for ir and it's still growing, I put more nad more
18:38 schestowitz-TR; in Daily Links
18:38 Techrights-sec; :/
21:59 schestowitz-TR; I've finally sorted out the microphone or sound quality on the new laptop
21:59 schestowitz-TR; but only after I had recoreded a couple with terrible quality
22:17 *u-amarsh04 (~amarsh04@joseon-rmogvn.g0d7.dtdf.mc4289.IP) has joined #boycottnovell
22:17 *u-amarsh04 (~amarsh04@v6xmmrhxmbafc.irc) has joined #boycottnovell
22:40 *u-amarsh04 has quit (connection closed)
22:40 *u-amarsh04 has quit (Connection closed)
23:16 schestowitz-TR; I sepent about an our trying to hack obs into doing webm as output
23:16 schestowitz-TR; I think due to a lack of CPU/GPU capacity this won't be possible
23:16 schestowitz-TR; as it cannot keep up
23:16 schestowitz-TR; unless it is sound only
23:57 psydruid; Intel/AMD won't work for the GPU offloading part?
-- Response ended
-- Page fetched on Sun May 19 15:45:11 2024