-- Leo's gemini proxy
-- Connecting to gemini.techrights.org:1965...
-- Connected
-- Sending request
-- Meta line: 20 text/gemini;lang=en-GB
01:29 *u-amarsh04 has quit (Quit: Konversation terminated!)
01:29 *u-amarsh04 has quit (Quit: Konversation terminated!)
01:33 *u-amarsh04 (~amarsh04@joseon-rmogvn.g0d7.dtdf.mc4289.IP) has joined #boycottnovell
01:33 *u-amarsh04 (~amarsh04@t3phqsdfxhjau.irc) has joined #boycottnovell
01:41 *psydroid4 has quit (Ping timeout: 2m30s)
04:50 *Despatche (~desp@u3xy9z2ifjzci.irc) has joined #boycottnovell
06:21 *SomeH4x0r has quit (Ping timeout: 2m30s)
07:45 Techrights-sec; If there is nothing more to test, I've concluded the experiment with the other
07:45 Techrights-sec; method of chat for now and close the tmux session on my end.
07:45 Techrights-sec; Almost no news coverage of Youtube-DL
07:45 Techrights-sec; also no mention of the constant demonetization of "Linux" videos
08:10 *DaemonFC has quit (Quit: Leaving)
08:45 schestowitz-TR; microsoft was very careful NOT to say why Friedman was "leaving"
08:45 schestowitz-TR; my guess is that they relied on many people not keeping abreast
08:45 schestowitz-TR; oif what was happening, and preferred to keep it that wayBBBAAA
08:46 Techrights-sec; yes, they often need people to be in the dark about what M$ and its minions
08:46 Techrights-sec; have been up to. PJ had some quote about that, something about the public
08:46 Techrights-sec; not being as stupid (ignorant) as microsoft needs them to be.
08:48 Techrights-sec; Also recall that M$ tried posting fake messages on other sites posing as PJ
08:48 Techrights-sec; in addition to MoG posting her home address along with admonishments to stop
08:48 Techrights-sec; by and kill her
08:48 schestowitz-TR; the quote from PJ was reprinted many times in TR
08:48 schestowitz-TR; but wait, I did not know about forgery of PJ
08:57 schestowitz; x https://cio.economictimes.indiatimes.com/news/digital-security/destructive-malware-targeting-ukrainian-govt-firms-microsoft/88946069
08:57 -TechrightsBN/#boycottnovell-cio.economictimes.indiatimes.com | microsoft: Destructive malware targeting Ukrainian govt, firms: Microsoft, CIO News, ET CIO
08:57 schestowitz; # m$ posing as an authority on the topic
09:02 schestowitz-TR; alert
09:02 schestowitz-TR; pi says file system has gone read-only
09:02 schestowitz-TR; investigating
09:03 schestowitz; "
09:03 schestowitz; Jan 17 04:44:17 raspberrypi kernel: [8538305.129032] EXT4-fs (mmcblk0p7): error count since last fsck: 31
09:03 schestowitz; Jan 17 04:44:17 raspberrypi kernel: [8538305.129047] EXT4-fs (mmcblk0p7): initial error at time 1642219644: ext4_read_inode_bitmap:199
09:03 schestowitz; Jan 17 04:44:17 raspberrypi kernel: [8538305.129056] EXT4-fs (mmcblk0p7): last error at time 1642388345: ext4_free_inode:352
09:04 *psydroid4 (~psydroid@cqggrmwgu7gji.irc) has joined #boycottnovell
09:04 Techrights-sec; ack
09:04 Techrights-sec; $ mount | grep -m1 '^/'
09:04 Techrights-sec; /dev/mmcblk0p7 on / type ext4 (ro,noatime)
09:05 schestowitz-TR; suggestions?
09:07 Techrights-sec; ack
09:07 Techrights-sec; thinking. was there an earlier error or one mentioning remounting as read-only?
09:10 Techrights-sec; filesystems are not my area. I might say power down the machine, remove the
09:10 Techrights-sec; microSD card, and fsck (without modiyfying) from another system to see what
09:10 Techrights-sec; if any errors there are.
09:10 schestowitz-TR; I see only one message in syslog saying it went R-O
09:11 schestowitz-TR; should I make a full backup before shutdown?
09:12 Techrights-sec; Do you have smartmontools available?
09:12 Techrights-sec; Backups are always good.
09:12 Techrights-sec; The contents of Gemini/bin are from Git the Gemini gemtext is important though
09:12 schestowitz-TR; I will rsync now
09:14 Techrights-sec; sudo smartctl -i /dev/mmcblk0 ?
09:15 schestowitz-TR; it is not installed and I cannot install anything
09:15 schestowitz-TR; meanwhile gemini server is still running OK
09:16 Techrights-sec; if the chip is removed it can be run on another system, don't try to write until
09:16 Techrights-sec; checking the microSD card a little
09:17 schestowitz-TR; rsync may run for hours for now, just to make sure I get all the latest things
09:19 Techrights-sec; ack
09:19 Techrights-sec; checked with a card here smartctl does not seem to deal with microSD
09:19 schestowitz-TR; after going RO the syslog file continued to expand for anotheer 6 minutes and then that's it
09:20 schestowitz-TR; my instinct was
09:20 schestowitz-TR; maybe reboot
09:20 schestowitz-TR; but that would not help find the underlying issuea
09:20 schestowitz-TR; and I don't know if the OS would fsck automatically upon reboot
09:23 Techrights-sec; I think it does but manually from another system might be more controllable
09:23 schestowitz-TR; I am not sure my other machines even have the right SD slot
09:24 Techrights-sec; No adapter?
09:24 Techrights-sec; Do you have a spare, good USB stick >= 16GB ?
09:25 schestowitz-TR; no, not stick but external hard drive, though I guess for booting a whole OS it needs formatting and I have lots of things on those already
09:26 Techrights-sec; I would recommemd burning Raspberry Pi OS Legacy to a USB stick and booting from
09:26 Techrights-sec; that if the start up cannot be resolved.
09:26 schestowitz-TR; yes, maybe a change to 'upgrade' a little, without dataloss
09:27 schestowitz-TR; notice it says this is not the first FS-related issue since last fsck
09:28 schestowitz; raspberrypi:/var/log# grep EXT4 syslog
09:28 schestowitz; Jan 17 02:31:39 raspberrypi kernel: [8530346.994847] EXT4-fs error (device mmcblk0p7): ext4_wait_block_bitmap:519: comm kworker/u8:2: Cannot read block bitmap - block_group = 36, block_bitmap = 1048580
09:28 schestowitz; Jan 17 02:31:39 raspberrypi kernel: [8530347.122079] EXT4-fs (mmcblk0p7): Delayed block allocation failed for inode 280973 at logical offset 0 with max blocks 69 with error 5
09:28 schestowitz; Jan 17 02:31:39 raspberrypi kernel: [8530347.122092] EXT4-fs (mmcblk0p7): This should not happen!! Data will be lost
09:28 schestowitz; Jan 17 02:58:14 raspberrypi kernel: [8531941.632828] EXT4-fs error (device mmcblk0p7): ext4_read_inode_bitmap:199: comm StreamT~s #6142: Cannot read inode bitmap - block_group = 51, inode_bitmap = 1572883
09:28 schestowitz; Jan 17 02:58:14 raspberrypi kernel: [8531941.954832] EXT4-fs error (device mmcblk0p7) in ext4_free_inode:352: IO failure
09:28 schestowitz; Jan 17 02:59:04 raspberrypi kernel: [8531992.227810] EXT4-fs error (device mmcblk0p7): ext4_read_inode_bitmap:199: comm firefox-esr: Cannot read inode bitmap - block_group = 51, inode_bitmap = 1572883
09:28 schestowitz; Jan 17 02:59:05 raspberrypi kernel: [8531993.258625] EXT4-fs error (device mmcblk0p7) in ext4_free_inode:352: IO failure
09:28 schestowitz; Jan 17 04:44:17 raspberrypi kernel: [8538305.129032] EXT4-fs (mmcblk0p7): error count since last fsck: 31
09:28 schestowitz; Jan 17 04:44:17 raspberrypi kernel: [8538305.129047] EXT4-fs (mmcblk0p7): initial error at time 1642219644: ext4_read_inode_bitmap:199
09:28 schestowitz; Jan 17 04:44:17 raspberrypi kernel: [8538305.129056] EXT4-fs (mmcblk0p7): last error at time 1642388345: ext4_free_inode:352
09:29 schestowitz; 24 hours earlier:
09:29 schestowitz; root@raspberrypi:/var/log# grep EXT4 syslog.1
09:29 schestowitz; Jan 16 04:42:30 raspberrypi kernel: [8451797.049016] EXT4-fs (mmcblk0p7): error count since last fsck: 26
09:29 schestowitz; Jan 16 04:42:30 raspberrypi kernel: [8451797.049026] EXT4-fs (mmcblk0p7): initial error at time 1642219644: ext4_read_inode_bitmap:199
09:29 schestowitz; Jan 16 04:42:30 raspberrypi kernel: [8451797.049034] EXT4-fs (mmcblk0p7): last error at time 1642268387: ext4_free_inode:352
09:29 Techrights-sec; yes
09:29 Techrights-sec; maybe a fsck would resolve those errors, but which files would it zap?
09:29 Techrights-sec; they are gone anyway but which ones?
09:30 schestowitz-TR; notice it says this is not the first FS-relateiirc, a fsck might give details of what files are affected, sometimesd issue since last fsck
09:31 Techrights-sec; yes and fsck can be done read-only if done manually , I'm not sure about the
09:31 Techrights-sec; boot process' fsck though
09:32 schestowitz-TR; as I understand it, the system already marked this card as needing fsck, but to access the system it needs to load it. anyway, let's finish rsync first, there's no downtime in the meantime, just a lack of capsule updates
09:32 schestowitz-TR; iirc, a fsck might give details of what files are affected, sometimes
09:34 schestowitz-TR; notice it says this is not the first FS-related issue since last fsck
09:36 Techrights-sec; A big question I have is about how to properly set up an underprovisioned
09:36 Techrights-sec; card under Raspberry Pi OS, since it seems to expand the file system to expand
09:36 Techrights-sec; to the whole device
09:36 Techrights-sec; rather than leaving a little extra for the wear leveling
09:38 activelow; nilfs2 isn't a bad choice (although it required some patches and extension for a fsck utility, which i had written some time ago)
09:38 activelow; nilfs2 implements wear-leveling naturally
09:38 schestowitz-TR; i realise zgrep is also installed, so:
09:39 schestowitz; /var/log# zgrep EXT4 syslog.2 | head -n 6
09:39 schestowitz; Jan 15 04:07:24 raspberrypi kernel: [8363290.618537] EXT4-fs error (device mmcblk0p7): ext4_read_inode_bitmap:199: comm StreamT~s #1069: Cannot read inode bitmap - block_group = 51, inode_bitmap = 1572883
09:39 schestowitz; Jan 15 04:07:24 raspberrypi kernel: [8363290.666938] EXT4-fs error (device mmcblk0p7) in ext4_free_inode:352: IO failure
09:39 schestowitz; Jan 15 08:31:05 raspberrypi kernel: [8379111.679073] EXT4-fs error (device mmcblk0p7): ext4_read_inode_bitmap:199: comm StreamT~s #1178: Cannot read inode bitmap - block_group = 51, inode_bitmap = 1572883
09:39 schestowitz; Jan 15 08:31:05 raspberrypi kernel: [8379111.727971] EXT4-fs error (device mmcblk0p7) in ext4_free_inode:352: IO failure
09:39 schestowitz; Jan 15 08:32:46 raspberrypi kernel: [8379213.143008] EXT4-fs error (device mmcblk0p7): ext4_read_inode_bitmap:199: comm Cache2 I/O: Cannot read inode bitmap - block_group = 51, inode_bitmap = 1572883
09:39 schestowitz; Jan 15 08:32:47 raspberrypi kernel: [8379214.084020] EXT4-fs error (device mmcblk0p7) in ext4_free_inode:352: IO failure
09:40 schestowitz-TR; activelow: on what media type?
09:40 schestowitz-TR; we wonder if maybe the pi should bot from external
09:40 schestowitz; root@raspberrypi:/var/log# zgrep EXT4 syslog.3
09:40 schestowitz; root@raspberrypi:/var/log# zgrep EXT4 syslog.4
09:40 schestowitz; root@raspberrypi:/var/log# zgrep EXT4 syslog.5
09:40 schestowitz; root@raspberrypi:/var/log# zgrep EXT4 syslog.6
09:40 schestowitz; root@raspberrypi:/var/log# zgrep EXT4 syslog.7
09:41 schestowitz; so it all seems to have started 48 hours ago. warning about data loss today, then read-only, several hours later
09:41 activelow; schestowitz-TR: on all media types
09:42 schestowitz; the issue here seems to be the physical media
09:42 schestowitz; not the file system itself
09:42 Techrights-sec; ack
09:43 activelow; which nilfs2 could have prevented
09:44 schestowitz-TR; generally speaking, all the data on the pi is not the 'master' copy, so even data
09:44 schestowitz-TR; loss does not cause me any panic
09:44 schestowitz-TR; given the limitations, e.g. not being able to fsk from another mahcine,
09:44 schestowitz-TR; maybe after rsync I will try a reboot
09:44 schestowitz-TR; if it boots OK, I will monitor syslog
09:44 Techrights-sec; ack
09:44 Techrights-sec; no panic, just inconvenience :(
09:44 Techrights-sec; ]
09:49 Techrights-sec; https://en.wikipedia.org/wiki/NILFS
09:49 Techrights-sec; That would require a lot of re-arrangement pre-install to partition the card
09:49 Techrights-sec; accordingly, afaik.
09:49 -TechrightsBN/#boycottnovell-en.wikipedia.org | NILFS - Wikipedia
09:49 Techrights-sec; $ apt-cache search nilfs
09:49 Techrights-sec; libguestfs-nilfs - guest disk image management system - NILFS v2 support
09:49 Techrights-sec; nilfs-tools - Continuous Snapshotting Log-structured Filesystem
09:49 Techrights-sec; It's there, however.
09:55 schestowitz; rsync: read errors mapping "/home/xxxxxxxxxxx/.ipfs/blocks/DM/CIQI7T5ZGALRVHRLN2JEPHEHK3KS46ALSXH3AINJ56NOM37A53UADMI.data": Input/output error (5)
09:55 schestowitz; rsync: read errors mapping "/home/xxxxxxxxxxx/.ipfs/blocks/DM/CIQI7T5ZGALRVHRLN2JEPHEHK3KS46ALSXH3AINJ56NOM37A53UADMI.data": Input/output error (5)
09:55 schestowitz; ERROR: xxxxxxxxxxx/.ipfs/blocks/DM/CIQI7T5ZGALRVHRLN2JEPHEHK3KS46ALSXH3AINJ56NOM37A53UADMI.data failed verification -- update discarded.
09:55 schestowitz-TR; upside of copying all of /home across
10:11 Techrights-sec; ack
10:22 schestowitz-TR; thus far no more file access issue detected, which is a relief
10:23 schestowitz; https://forums.raspberrypi.com/viewtopic.php?t=88429
10:23 -TechrightsBN/#boycottnovell-forums.raspberrypi.com | how to fsck the boot SD card? - Raspberry Pi Forums
10:24 Techrights-sec; very good
10:27 Techrights-sec; It looks like the ability to read/write the microSD card from another machine
10:27 Techrights-sec; is necessary.
10:29 schestowitz-TR; I have sd card slot/s, but afaikk not microsd, but can go to town for adapter if
10:29 schestowitz-TR; needed
10:29 Techrights-sec; Wasn't there an adapter in the kit?
10:30 schestowitz-TR; don't think I saw an adapter. The only unused bit is the buttons, which need soldering
10:33 *tech_exorcist (~tech_exorcist@d3u7abs8yy2fu.irc) has joined #boycottnovell
11:00 Techrights-sec; ack
11:00 Techrights-sec; new cards are available with an adapter, though I'm wondering if a USB stick
11:00 Techrights-sec; would be better for the (eventual) ssytem upgrade: it would be faster and
11:00 Techrights-sec; more wear resistent.
11:23 schestowitz-TR; ok, rsync is now done
11:23 schestowitz-TR; the only errors were some transident blocks of ipfs
11:23 schestowitz-TR; the ones I listed before
11:23 schestowitz-TR; or rather,
11:23 schestowitz-TR; it seems to bejust onme
11:23 schestowitz-TR; that same one named thrice
11:23 schestowitz-TR; those files can be rebuilt, I think, thiunnk of them as cache
11:23 schestowitz-TR; maybe they are the ones that triggered all the error
11:24 Techrights-sec; ok, can those files be reconstructed?
11:24 schestowitz-TR; good idea to reboot and, if so, what sequence? would sudo reboot even work?
11:24 Techrights-sec; yes sudo reboot will work but the question is what will happen when the system
11:24 Techrights-sec; tries to start again
11:25 schestowitz-TR; I doubt it will get any worse if rebooted, as it would stop if there's an FS panic
11:25 Techrights-sec; ok
11:25 schestowitz-TR; rebooting
11:26 Techrights-sec; ack
11:26 Techrights-sec; be sure to reboot the right machine
11:30 schestowitz-TR; ok, on boot screen
11:30 schestowitz-TR; "failed to open device: "sdcard"
11:30 schestowitz-TR; and it loops as it re-attempts
11:31 Techrights-sec; be sure to reboot the right machine
11:31 Techrights-sec; ack
11:31 Techrights-sec; an adapter will be needed to work on it from another machine with fsck or
11:31 Techrights-sec; something. A spare and maybe a USB stick miight come in handy
11:34 Techrights-sec; IPFS and Geminni workflows have been write-intensive.
11:34 Techrights-sec; Ok. The migration to USB stick is not so hard. Given the resource usage
11:34 Techrights-sec; at least 16GB is needed. I'll try some stuff here today about the under
11:34 Techrights-sec; provisioning.
11:34 schestowitz-TR; should we not use this opportunity to move to another media type and maybe OS?
11:34 Techrights-sec; Do you want to stay with Raspbery Pi OS, if so legacy or Bullseye
11:34 Techrights-sec; or move to Devuan?
11:37 Techrights-sec; let me put it like this
11:37 Techrights-sec; 1. we might have a defective card
11:37 Techrights-sec; so we might need to reinstall regardless
11:37 Techrights-sec; 2. we have a change to migrate or upgrade
11:37 Techrights-sec; 2a. storage
11:37 Techrights-sec; 2b. OS
11:37 Techrights-sec; Based on your experience and psydroid, microsd is prone to such incidents
11:37 schestowitz-TR; let me put it like this
11:37 Techrights-sec; so I am leaning towards moving away from this volatility
11:37 schestowitz-TR; 1. we might have a defective card
11:37 schestowitz-TR; so we might need to reinstall regardless
11:37 schestowitz-TR; 2. we have a change to migrate or upgrade
11:37 schestowitz-TR; 2a. storage
11:37 schestowitz-TR; 2b. OS
11:37 schestowitz-TR; Based on your experience and psydroid, microsd is prone to such incidents
11:37 schestowitz-TR; so I am leaning towards moving away from this volatility
11:38 Techrights-sec; There are a lot of ways a bloc kwithin the card can wear out.
11:38 Techrights-sec; Migration of medium would be appropriate at this time.
11:38 Techrights-sec; Yes, they wear out, plus the market is flooded with lowquality knockoffs.
11:38 Techrights-sec; another advantage of USB is that the adapter is not needed
11:39 schestowitz-TR; ok, let me think
11:39 schestowitz-TR; I have a very old magnetic usk disk, usb2 I thjink, from 2007 ish
11:39 schestowitz-TR; I have not ueven used it for ages
11:40 Techrights-sec; A new USB stick would be advised, if there is a budget. The 16GB or 32GB
11:40 Techrights-sec; ought to be inexpesive
11:41 schestowitz-TR; ok, bear me me
11:41 schestowitz-TR; I have the capacity to go to town, might grab food as we need some
11:41 schestowitz-TR; would; you suggest I pourchase a standard USB stick 32 GB in size and
11:41 schestowitz-TR; m, if so, how hard is the setup process?
11:41 schestowitz-TR; I am 2 days off work now, so time is not an issue, I can leave
11:41 schestowitz-TR; the github series on the ice for now
11:49 Techrights-sec; That would work. The process is as described the other day,
11:49 Techrights-sec; download the RPiOS image and use dd to put it on the stick, boot from the stick,
11:49 Techrights-sec; and restore the system and then the data. A microSD adapter might be good to
11:49 Techrights-sec; have in the toolbox, I thought one was available in the kit.
11:49 Techrights-sec; Looking online the sticks should be in the vicinity of 8 pounds
11:49 Techrights-sec; or less
11:49 Techrights-sec; Which OS / version ?
12:00 schestowitz-TR; i AM SEARCHING TGe web for some stuff, worse than useless
12:00 schestowitz-TR; s/n ratio even in Gulag Search
12:00 schestowitz-TR; Do we know for sure this model will boot OK from USB stick and detect it OK?
12:01 Techrights-sec; RPi4 does boot from USB, the question is does it need changing settings first
12:01 Techrights-sec; probably not. The RP3B+ also boots from USB.
12:02 schestowitz-TR; I have not attached a keyboard, but would I ned to access soime boot menu?
12:02 Techrights-sec; If settings are changed, it happens via raspi-config. So in the worst case
12:02 Techrights-sec; set up another microSD card boot that and then change the settings then boot
12:02 Techrights-sec; from USB
12:03 schestowitz-TR; oh, dear, so I mighr even need to buy a card, install an OS, just to tell the system to boot from USB?
12:03 Techrights-sec; in the worst case, I don't have an unmodified system to try with
12:03 Techrights-sec; Howver, it is /supposed/ to boot from USB
12:04 schestowitz-TR; ok, let me experiment to see how empty sd slot make the pi behave, or plug some j
12:04 schestowitz-TR; unk usb device
12:05 Techrights-sec;
12:05 Techrights-sec; If moving to Bullseye:
12:05 Techrights-sec; https://downloads.raspberrypi.org/raspios_armhf/images/raspios_armhf-2021-11-08/2021-10-30-raspios-bullseye-armhf.zip.torrent
12:05 Techrights-sec; Otherwise:
12:05 Techrights-sec; https://downloads.raspberrypi.org/raspios_oldstable_armhf/images/raspios_oldstable_armhf-2021-12-02/2021-12-02-raspios-buster-armhf.zip.torrent
12:09 schestowitz-TR; "USB MSD timed out after 20 seconds"
12:09 schestowitz-TR; when no sd card inserted
12:09 schestowitz-TR; Not sure what MSD is
12:09 schestowitz-TR; it goes like this in a loop
12:09 schestowitz-TR; does it mean it checks both USB and SD?
12:14 Techrights-sec; it checks the SD first, by default, so the card has to be removed
12:14 Techrights-sec; to test the USB boot. if it boots from USB then raspi-config can be used
12:14 Techrights-sec; to have it try USB first the failover to SD if necessary
12:14 schestowitz-TR; ok, I have just inserted an old usb stick
12:14 schestowitz-TR; it is at l;east attempting to load from it
12:14 schestowitz-TR; failinging, obviously
12:15 Techrights-sec; ok, then can you burn one of the above images, or Devuan, to the stick and
12:15 Techrights-sec; try that way?
12:16 schestowitz-TR; this stick is 256 MB(yes, MB), so I need to buy one
12:17 Techrights-sec; Ok that's too small for anything relevant here.
12:20 schestowitz-TR; so the plan is to buy a USB stick, put on it some OS
12:20 schestowitz-TR; (not sure which) and then create the account again and restore tom state similar
12:20 schestowitz-TR; to the prior state?
12:21 schestowitz-TR; be back in 10 minutes, there is a store near us that MIGH have USB sticks...
12:31 schestowitz-TR; back
12:31 schestowitz-TR; they don\t hgave it
12:31 Techrights-sec; ack
12:34 schestowitz-TR; psydroid4: ping
12:34 schestowitz-TR; thoughts?
12:34 schestowitz-TR; what you try toi 'fix' the nicroSD?
12:35 psydroid4; schestowitz-TR, I didn't do anything, I just had some spare cards lying around from years ago
12:35 schestowitz-TR; microsd?
12:35 psydroid4; yes
12:35 schestowitz-TR; so you think using microSD again is worth it?
12:35 psydroid4; no, I don't actually think so
12:36 schestowitz-TR; because i might go to town for USB stick instead, assuming it's not going to die like this sandisk card after 14 months of use
12:36 psydroid4; I am only using the microSD card for u-boot, the dtb file and the kernel
12:36 psydroid4; the / is on a USB SATA SSD
12:36 schestowitz-TR; I see
12:36 schestowitz-TR; it's possible to boot from USB, right?
12:36 schestowitz-TR; raspi4
12:37 psydroid4; I think your Raspberry Pi 4 firmware should be able to boot directly from USB, if you change that one setting
12:37 psydroid4; yes
12:37 psydroid4; it is
12:37 schestowitz-TR; assuming I cannot boot from microSD again, would it be fine to start everything ferom USB?
12:37 schestowitz-TR; I experiemented weith an invalud usb
12:38 schestowitz-TR; at least iut tries it
12:38 psydroid4; I had to set a 10s delay for mine for the kernel to be able to find /dev/sda1
12:38 psydroid4; yes, I think that would be fine
12:38 schestowitz-TR; ok, thanks
12:38 psydroid4; it's actually much better, even if that's not the most aesthetic solution
12:40 activelow; schestowitz-TR: with many other ARM SBC (except raspberry) it is easier to load an operating system from microsd
12:40 schestowitz-TR; i just don't trust them anymore
12:41 schestowitz-TR; psydroid4: , Techrights-sec and I seem to have issues after a year
12:41 schestowitz-TR; recovery takes ages even with backup if the setup is complex
12:41 psydroid4; they are just not made for constant writes
12:42 activelow; without an appropriate filesystem (nilfs2, f2fs) failure of microsd is guaranteed; question is not IF these fail, question is how quickly those fail
12:42 schestowitz-TR; i wondr if it is time to move ipfs to a datacentre too
12:43 schestowitz-TR; for bandwidth
12:43 activelow; probably not, create regular backups, replace the sdcard, and use an appropriate filesystem
12:43 schestowitz-TR; not USB|?
12:44 activelow; depends, USB contains "flash memory" too, which is prone to write amplification, depends on the controller
12:45 activelow; USB complicates things; microsd "flash memory" is cheap, ~4.5Euros for 16GiB Verbatim brand, which lasts a while with an approprieate file system
12:46 schestowitz-TR; maybe get a microsd adapter?
12:46 schestowitz-TR; to see if I can repair it from a laptop with fsck?
12:46 schestowitz-TR; and also buy a spare microsd?
12:46 activelow; i had to dispose *many* microsd some time ago; for a yet unknown reason i might add, because these weren't stressed with many writes
12:46 schestowitz-TR; i would need an adapter for it regardless to write the OS
12:47 activelow; in any case, backups, and a flash-friendly filesystem, then microsd are an economical choice
12:47 schestowitz-TR; OK
12:49 schestowitz-TR; psydroid4: would that not still leave the issue of frequent dead OS?
12:49 schestowitz-TR; compared to USB?
12:49 psydroid4; schestowitz-TR, I run everything from USB, also the operating system
12:49 schestowitz-TR; iit says trhe adapter is like 3 pounds
12:50 schestowitz-TR; psydroid4: so is it more robust?
12:50 psydroid4; schestowitz-TR, it's much more robust in my experience
12:50 psydroid4; I've been running like this for a year now
12:50 schestowitz-TR; activelow cautions that this too has a sensitivity to write-intensive usage
12:50 schestowitz-TR; maybe i will get microsd, adapter, and also usb
12:51 psydroid4; what is important with flash media is to not fill them up completely and also to set things up so as to reduce writes
12:52 schestowitz-TR; thanks, will go buyu things now
12:53 psydroid4; you can use "noatime" and "nodiratime" as mount options
12:53 psydroid4; I still have an issue of excessive writes to logs, so I need to look into that
12:53 psydroid4; maybe I enabled debug somewhere
12:54 activelow; schestowitz-TR: i said "write-amplification"; choose an appropriate filesystem such as nilfs2 or f2fs which are "flash friendly"
12:56 schestowitz-TR; i am going to buy
12:56 schestowitz-TR; 1. sd card reader to usb
12:56 schestowitz-TR; 2. usb stick
12:56 schestowitz-TR; 3. spare sd card
12:57 Techrights-sec; the microSD cards often come with adapters for normal SD
12:57 Techrights-sec; so if ytou have a normal sized SD slot, that would be all that is needed.
12:57 Techrights-sec; THe are only marginally more expensive, online they are 0.50 Euro but ordering
12:57 Techrights-sec; online is out of the question at the moment.
12:57 schestowitz-TR; oh, ok
12:58 *psydroid4 is really running from a USB SSD, not just a USB flash drive
12:59 psydroid4; https://m.media-amazon.com/images/I/61w6Rs68uLS._AC_SY450_.jpg
13:00 psydroid4; and then a SATA SSD connected to it
13:10 *u-amarsh04 has quit (Quit: Konversation terminated!)
13:10 *u-amarsh04 has quit (Quit: Konversation terminated!)
13:19 *u-amarsh04 (~amarsh04@joseon-rmogvn.g0d7.dtdf.mc4289.IP) has joined #boycottnovell
13:19 *u-amarsh04 (~amarsh04@t3phqsdfxhjau.irc) has joined #boycottnovell
14:11 *SomeH4x0r (~someh4xx@kxgbrhbcjzafi.irc) has joined #boycottnovell
14:32 schestowitz-TR; ok, back home
14:32 schestowitz-TR; got 32gb sandisk microsd with sd adapter
14:33 schestowitz-TR; it was 15 pounds, compared to some cheaper one of the same specs at 4 pounds
14:33 schestowitz-TR; too expensive to lose an OS
14:33 schestowitz-TR; I also bought one USB stick 32GB
14:33 schestowitz-TR; plan is, firts test the faulty microSD and see if I can mount te repaid the file system
14:53 *psydroid4 has quit (Quit: Leaving)
14:53 *psydroid4 (~psydroid@cqggrmwgu7gji.irc) has joined #boycottnovell
14:58 schestowitz-TR; :/var/log# fsck /dev/sdb
14:58 schestowitz-TR; fsck from util-linux 2.20.1
14:58 schestowitz-TR; e2fsck 1.42.9 (4-Feb-2014)
14:58 schestowitz-TR; fsck.ext2: No medium found while trying to open /dev/sdb
14:58 schestowitz-TR; The superblock could not be read or does not describe a valid ext2/ext3/ext4
14:58 schestowitz-TR; filesystem. If the device is valid and it really contains an ext2/ext3/ext4
14:58 schestowitz-TR; filesystem (and not swap or ufs or something else), then the superblock
14:58 schestowitz-TR; is corrupt, and you might try running e2fsck with an alternate superblock:
14:58 schestowitz-TR; e2fsck -b 8193 <device>
14:58 schestowitz-TR; or
14:58 schestowitz-TR; e2fsck -b 32768 <device>
14:58 schestowitz-TR; (that's with the old card inserted)
14:58 schestowitz-TR; (the new microsd mounts ok)
14:59 schestowitz-TR; (but would prefer to salvage what we have)
15:00 schestowitz-TR; [482680.138495] sd 8:0:0:0: [sdb] CDB:
15:01 schestowitz-TR; [482680.138503] Read(10): 28 00 00 00 00 00 00 00 08 00
15:01 schestowitz-TR; [482680.138552] end_request: I/O error, dev sdb, sector 0
15:01 schestowitz-TR; [482680.138563] Buffer I/O error on device sdb, logical block 0
15:01 schestowitz-TR; [482680.138593] sdb: unable to read partition table
15:01 schestowitz-TR; [482680.139177] sd 8:0:0:0: [sdb] Attached SCSI removable disk
15:01 schestowitz-TR; [482760.036978] tpm_tis tpm_tis: command 0x65 (size 20) returned code 0x0
15:01 schestowitz-TR; [482760.066940] tpm_tis tpm_tis: command 0x65 (size 22) returned code 0x0
15:01 schestowitz-TR; [482760.097191] tpm_tis tpm_tis: command 0x65 (size 22) returned code 0x0
15:03 schestowitz; looking at https://raymii.org/s/blog/Broken_Corrupted_Raspberry_Pi_SD_Card.html
15:03 -TechrightsBN/#boycottnovell-raymii.org | Broken Corrupted Raspberry Pi SD Card - Raymii.org
15:04 Techrights-sec; can the old microsd be put in the adapter and examined in another machine?
15:04 schestowitz-TR; this is what I am doing, trying to repair the broken card on another machine
15:09 Techrights-sec; ack
15:10 schestowitz-TR; you were right to suggest that I keep a spare card, but at that point I already stopped going to town except for food
15:10 schestowitz-TR; so far, all the suggestions in that page have not helped
15:11 Techrights-sec; checking
15:11 schestowitz-TR; at this point I still try to salvage. failing that, we have usb stick or microsd that works, each of them 32gb
15:24 schestowitz-TR; activelow / psydroid4 your advice would be appreciated
15:24 schestowitz-TR; a lot of services run on this machine
15:26 psydroid4; schestowitz-TR, I was never able to save my files, but I had created a clone of the installation on another microSD card
15:27 psydroid4; I learned the hard way that microSD cards are just not reliable for long-term storage of important files
15:27 schestowitz-TR; would you suggest I use the USB stick?
15:27 schestowitz-TR; I got 32GB microsd and 3gb usb stick
15:27 psydroid4; it's not about USB stick vs microSD card, I think
15:27 schestowitz-TR; I thought in case this card is dead it would help to have a spare anyway
15:28 psydroid4; and I'm afraid the USB stick might die in a very similar way in the future
15:28 psydroid4; I use them when I need mostly read-only media with relatively few writes
15:29 schestowitz-TR; opk, thanks
15:29 schestowitz-TR; based on some forums, when I cannot even get the file system types and just the device itself it likely means the end of the cord
15:30 psydroid4; the USB<->SATA cable of which I posted a picture before cost me just 4 and I had this SATA SSD that cost me 19
15:30 schestowitz-TR; i get sdb
15:30 schestowitz-TR; but never sdb1 and etc.
15:30 schestowitz-TR; even tools like fdisk cannot decipher anything based on the superblock
15:31 psydroid4; in the long term it will be cheaper to go with my solution
15:31 psydroid4; yes, I tried everything with mine too
15:31 psydroid4; I couldn't even read files off it, because the device couldn't be found or mounted anymore
15:35 schestowitz-TR; ok, should I install the latest debian 11-based OS on the new card?
15:39 Techrights-sec; Debian or Devuan but then there would be the question of Blinkt. I'll look a lit
15:39 Techrights-sec; tle to see if I can find anything about that. Seems to be available,
15:39 Techrights-sec; at the very least via pip3 https://pypi.org/project/blinkt/
15:39 -TechrightsBN/#boycottnovell-pypi.org | blinkt PyPI
15:39 Techrights-sec; But can't seem to find any packages in the official Debian repositories
15:39 schestowitz-TR; I assume "legacy" means Debian Buster
15:40 psydroid4; how did you install Debian on it previously?
15:40 psydroid4; I am running Debian 11 on mine
15:40 schestowitz-TR; the raspi came with debian-based raspi OS on the sd card
15:42 Techrights-sec; Yes it is Buster with modifications
15:42 Techrights-sec; dd can be used, see the torrent links previously
15:42 psydroid4; I would install Debian rather than Raspi OS
15:43 psydroid4; but I see that sound may not be working on the Debian wiki
15:43 psydroid4; so I don't know how important that is
15:45 schestowitz-TR; not so important in my case, at least for now
15:45 schestowitz-TR; if I get rasp os lite, which is a zip file, I will need to check how to put it on the card
15:46 schestowitz-TR; it's 463mb, I assume I can add all the bits to it later, inc. gui
15:52 Techrights-sec; the image comes as compressed .zip file which can be expanded using unzip
15:52 Techrights-sec; then it can be transferred to the tarrget microSD card using dd, just
15:52 Techrights-sec; be very careful to verify the target device
15:52 Techrights-sec; Then once it is burned, it can be mounted like any other device and you can
15:52 Techrights-sec; so some minimal setup that way. However while you have SSH forwarded from
15:52 Techrights-sec; the outside, don't turn SSH on until after you've changed the password fro
15:52 Techrights-sec; the pi account. If you want to pre-load the SSH server's keys, you can do that
15:52 Techrights-sec; but the file
15:52 Techrights-sec; ./etc/systemd/system/multi-user.target.wants/regenerate_ssh_host_keys.service
15:52 Techrights-sec; needs to be removed or the keys will be erased on first boot
15:52 Techrights-sec; the sshd_config file is left alone even during first boot so it does not need
15:52 Techrights-sec; any extra effort
16:01 Techrights-sec; the lite edition will boot headless, the GUI can be added later but
16:01 Techrights-sec; if I recall correctly the lite edition had trouble booting from USB
16:01 schestowitz-TR; downloading standrad with desktop now
16:07 schestowitz-TR; wait
16:07 schestowitz-TR; you mention USB
16:07 schestowitz-TR; do you want to reun it from external USB stick or microsd?
16:07 schestowitz-TR; both are 32GB
16:13 Techrights-sec; which is your preference? I have moved one of mine to USB when a microSD
16:13 Techrights-sec; card died recently. However, it may not matter much.
16:13 schestowitz-TR; the sd card is the same make as the last one (not cheap!) but the USB is c cheaper chinese brand, so I am leaning against it
16:14 schestowitz-TR; upside is, we get OS upgrade and also twice as much disk space
16:15 Techrights-sec; ok in that case maybe the microSD card is the way to go, but consider it
16:15 Techrights-sec; something that will need replacing and prepare contingency plans so that
16:15 Techrights-sec; it is less of a surprise in the future
16:55 *wallacer has quit (Ping timeout: 2m30s)
17:00 *wallacer (~quassel@6bsu33ajs4zs4.irc) has joined #boycottnovell
17:08 schestowitz; Jan 17 17:05:44 vonick kernel: [6398187.040038] pcieport 0000:00:1c.3: AER: Corrected error received: 0000:00:1c.3
17:08 schestowitz; Jan 17 17:05:44 vonick kernel: [6398187.040088] pcieport 0000:00:1c.3: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
17:08 schestowitz; Jan 17 17:05:44 vonick kernel: [6398187.040100] pcieport 0000:00:1c.3: device [8086:9d13] error status/mask=00000001/00002000
17:08 schestowitz; Jan 17 17:05:44 vonick kernel: [6398187.040109] pcieport 0000:00:1c.3: [ 0] RxErr (First)
17:08 schestowitz; on my main laptop every 10 mins or so today
17:08 schestowitz; is this something to worry about? didn't check syslog before, but only just noticed this
17:10 Techrights-sec; not sure. If it is a regular hard drive you can install smartmonctl and
17:10 Techrights-sec; use that to investigate; it should read the firmware are report on
17:10 Techrights-sec; its status
17:11 Techrights-sec; s/smartmonctl/smartcl/ its part of smartmontools packag
17:11 schestowitz-TR; can I dd the image from a usb stick to the microsd?
17:11 schestowitz-TR; this machine is very spoace-tight and the img file is 3.7 gb
17:12 Techrights-sec; best to go straight from the file and then erase the img file after first
17:12 Techrights-sec; successful boot IMO
17:12 schestowitz-TR; wait
17:12 schestowitz-TR; isn't dd just copying the img to the card?
17:12 schestowitz-TR; is there another way?
17:14 Techrights-sec; There might be but I have not tried using dd from one device to another like
17:14 Techrights-sec; that.
17:14 schestowitz-TR; i mean, usb:/some_dor/some image file dd this to /microsd
17:15 Techrights-sec; That's what I mean,
17:15 Techrights-sec; e.g. dd status=progress bs=1M if=/dev/sdc of=/dev/mmcblk0
17:15 Techrights-sec; or something similar
17:16 Techrights-sec; I can try here but it can take some tens of minutes to shuffle hardware,
17:16 Techrights-sec; just a sec...
17:35 Techrights-sec; i'm not sure the method is worth trying, if there are slight differences in
17:35 Techrights-sec; the size of the two devices there will be a mismatch. If the target is slightly
17:35 Techrights-sec; larger then the source, it is not so much of a problem. If the target is even
17:35 Techrights-sec; slightly smaller than the source, then the result is a broken file system
17:35 Techrights-sec; as far as I know. So someone who actually knows something about file systems
17:35 Techrights-sec; would have to comment. I'd recommend just burning twice, once each to USB and
17:35 Techrights-sec; microSD and then recovering the space.
17:35 Techrights-sec; There was a research article from the late 1990s about how faster computers
17:35 Techrights-sec; were a effectively a force multiplier. One loses a lot of time fiddling.
17:35 Techrights-sec; it makes sense but the /dev/sdb is probably your hard drive and runnning that
17:35 Techrights-sec; will erase it. Triple check the destination. The microSD card will be some
17:35 Techrights-sec; kind of mmcblk0 or mmcblk1 or similar.
17:35 Techrights-sec; Plug in the device and then check what the kernel reports:
17:35 Techrights-sec; dmesg | tail
17:35 Techrights-sec; I usually check dmesg immediately after plugging (or replugging) the device in
17:35 Techrights-sec; ack
17:35 schestowitz-TR; ack
17:35 schestowitz-TR; i am zeoring the sd card at the moment
17:35 schestowitz-TR; nezt, sudo dd bs=4MB if=usb path to image file of=/dev/sdb oflag=sync
17:35 schestowitz-TR; does that make sense?
17:35 schestowitz-TR; sdb 8:16
17:35 schestowitz-TR; 1 29.7G 0 disk
17:35 schestowitz-TR; sdb1 8:17
17:35 schestowitz-TR; 1 29.7G 0 part /media/removable/SD Card
17:35 schestowitz-TR; from lablk
17:35 schestowitz-TR; i xhwxkws dmesg also
17:35 schestowitz-TR; checked
17:36 schestowitz-TR; this is not a standard machine
17:36 schestowitz-TR; how long should the zeroing run roughly?
17:41 Techrights-sec; The zeroing only needs a few megabytes to do its job.
17:41 Techrights-sec; dd if=/dev/zero of=/dev/xxxxx bs=512 count=1
17:41 Techrights-sec; that is the minimum to clear the partition table
17:41 schestowitz-TR; dd bs=4M if=/dev/zero of=/dev/sdb oflag=sync
17:41 schestowitz-TR; almost 10 mins now
17:42 schestowitz-TR; that device is not mounted, which I suppose is the right thing to do
17:42 Techrights-sec; yes, it should be unmounted while erasing
17:43 schestowitz-TR; top shows dd going at 2% of CPU. maybe many write operations take a while and str
17:43 schestowitz-TR; art this card with wear and tear?
17:46 *SomeH4x0r has quit (Ping timeout: 2m30s)
17:51 *SomeH4x0r (~someh4xx@fjydu32baevf2.irc) has joined #boycottnovell
17:52 Techrights-sec; some but it does just one pass the wear leveling is managed within the
17:52 Techrights-sec; device itself. IF you can get the Raspberry Pi OS to treat it as less
17:52 Techrights-sec; than 32GB, leaving the rest for spare, the wear leveling routines will
17:52 Techrights-sec; draw on that reserve to extend the life of the active sections.
18:08 schestowitz-TR; dd: error writing /dev/sdb: No space left on device
18:08 schestowitz-TR; 7610+0 records in
18:08 schestowitz-TR; 7609+0 records out
18:08 schestowitz-TR; 31914983424 bytes (32 GB) copied, 2336.55 s, 13.7 MB/s
18:08 schestowitz-TR; i asssume this is OK
18:08 Techrights-sec; it's ok with /dev/zero as the source, but if something else was the source
18:08 Techrights-sec; then it is not quite ok
18:12 schestowitz-TR; writing the image now
18:12 schestowitz-TR; dd bs=4MB if=/var/host/media/removable/KIOXIA/2021-10-30-raspios-bullseye-armhf.i
18:12 schestowitz-TR; mg of=/dev/sdb oflag=sync
18:12 Techrights-sec; ack
18:19 schestowitz-TR; 993+1 records in
18:19 schestowitz-TR; 993+1 records out
18:19 schestowitz-TR; 3972005888 bytes (4.0 GB) copied, 315.045 s, 12.6 MB/s
18:28 Techrights-sec; If you copy over the host keys for SSHd prior to booting then,
18:28 Techrights-sec; ./etc/systemd/system/multi-user.target.wants/regenerate_ssh_host_keys.service
18:28 Techrights-sec; file has to be removed
18:30 *SomeH4x0r has quit (Ping timeout: 2m30s)
18:32 schestowitz-TR; good news
18:32 schestowitz-TR; changed pi password
18:32 schestowitz-TR; updating system now
18:32 schestowitz-TR; will soon
18:32 schestowitz-TR; start restoring services
18:33 schestowitz-TR; gemini likely first
18:33 schestowitz-TR; what best way to restort user accounts?
18:33 schestowitz-TR; just copying /home in for all non-pi accounts would not work well, then there's t
18:33 schestowitz-TR; he permission issue
18:45 *SomeH4x0r (~someh4xx@r8dui6smnhchc.irc) has joined #boycottnovell
19:00 schestowitz-TR; copying gemini from backup to gemini acccount
19:00 schestowitz-TR; the approach is,
19:00 schestowitz-TR; create each account in turn
19:00 schestowitz-TR; then pull files from it
19:00 schestowitz-TR; to maintain poermissions and owner
19:00 Techrights-sec; rsync -avH
19:00 Techrights-sec; then spot check ownership
19:02 Techrights-sec; if the non-pi accounts are created in the same order as they were the first
19:02 Techrights-sec; time then the numeric UID is usually the same but with chown -R that does not mat
19:02 Techrights-sec; ter so much. There probably aren't any files within the home direcories which
19:02 Techrights-sec; have non-standard ownership
19:03 schestowitz-TR; OK, in case of permissions issues I can always wire and re-fetch
19:03 schestowitz-TR; I was already running without rsync options
19:03 schestowitz-TR; not sure if running it again would correct or alter things
19:03 Techrights-sec; There is also a --usermap option in rsync
19:03 Techrights-sec; chown is the main question
19:04 schestowitz-TR; I just decided to restore thns in order of urgency/importance, not thinking that, given the OS change, oirder of user creation would matter
19:04 schestowitz-TR; we might also want to follow activelow advice on file system
19:05 schestowitz-TR; 1.9gb of gemini copied already, still going
19:05 Techrights-sec; probably but has the card been resized ? It would, I think, be possible to set
19:05 Techrights-sec; up a third partition for the data using gparted and then move /home to it,
19:05 Techrights-sec; after formatting.
19:06 schestowitz-TR; partitioning is not part of the installer, but you can hop on in soon (ssh is open, did not check from outside) when the account is back
19:06 schestowitz-TR; I still need to check the key thing, got it in notes, but am treating it as a lesser urgent thing
19:10 Techrights-sec; Bringing over the old host keys saves a lot of headache
19:11 schestowitz-TR; created 5 accounts, will pull in files one at a time, as each account in turn
19:17 schestowitz-TR; restorting 3 accounts at once over ssh
19:17 schestowitz-TR; later will do cronjabs
19:18 schestowitz-TR; ipfs and leds etc mighyt be hardest and least urgent
19:18 Techrights-sec; ack
19:31 schestowitz-TR; try ssh to pi
19:31 schestowitz-TR; keep hopes low, for now...
19:34 Techrights-sec; ok
19:34 Techrights-sec; key needs fixing, that's easy, populate /etc/ssh/ and then systemctl restart ssh
19:34 schestowitz-TR; would existing (live) connections be dropped?
19:36 Techrights-sec; I don't think so, but tmux will make that not a problem anyway
19:36 Techrights-sec; just a sec, testing
19:36 Techrights-sec; existing connections seem to be retained even after using systemctl restart ssh
19:47 schestowitz-TR; try now
19:49 Techrights-sec; Host key verification failed.
19:50 schestowitz-TR; can you remind me if pi account too needs something? when I remotely re--access the machine with any account I need to discard the entry in ..ssh firsdt
19:52 schestowitz-TR; pi account will be the trickier one to merge as it has some system-specific bulls
19:52 schestowitz-TR; eye stuff, so I need to be selective what I merge in and how
20:38 *tech_exorcist has quit (Quit: bbl)
20:42 schestowitz-TR; I have just got gemini running correctly agai, agate changed the way
20:42 schestowitz-TR; this is done in their new version and I run it manually as I could
20:42 schestowitz-TR; not find a service for agate, OI hope we have backups of these
20:42 schestowitz-TR; configs somewhere
20:42 *tech_exorcist (~tech_exorcist@b3rs3wwrk3aiu.irc) has joined #boycottnovell
20:44 schestowitz-TR; to sort out your ssh keys you can log in with the password
20:59 *tech_exorcist has quit (Quit: Disconnecting)
21:53 schestowitz; > Can't seem to get the Gemini pages today.
21:53 schestowitz; > cheers,
21:53 schestowitz; The Raspi broke down. Actually, the storage went all bad. After spending ours trying to salvage it (and getting a recent enough backup) I went to the store, bought a bunch of things (USB stick, adapter, new MicroSD) and worked to recover IPFS, Gemini, IRC and loads of other things running on this machine. Recovery is still work in progress, but Gemini is generally back now. I've long dreaded the moment of breakdown; MicroSD cards don't last long
21:53 schestowitz; if a whole OS runs on them.
21:53 schestowitz; Upside is: twice as much storage space and upgrade from Debian 10 to 11 (actually clean install).
21:53 schestowitz; Maybe I'll write about it soon, but for a long story the IRC logs are there. Esp. the #boycottnovell channel (17-01-22).
22:24 *blitzed has quit (Quit: +++ATH0&D2 NO CARRIER NO CAREER)
22:51 *psydroid4 has quit (Ping timeout: 2m30s)
23:32 *activelow has quit (Ping timeout: 2m30s)
23:48 *activelow (~activelow@254g4tq7df99e.irc) has joined #boycottnovell
-- Response ended
-- Page fetched on Sun May 19 15:06:02 2024
