Lessons learned with XZ vulnerability

Author: Solène

Date: 30 March 2024

Tags: security linux openbsd

Comment on Mastodon

Intro

Yesterday Red Hat announced that xz library was compromised badly, and could be use as a remote execution code vector. It's still not clear exactly what's going on, but you can learn about this on the following GitHub discussion that also links to original posts:

Discussion about xz being compromised

What's the state?

As far as we currently know, xz-5.6.0 and xz-5.6.1 contains some really obfsucated code that would trigger only in sshd, this only happen in the case of:

the system is running systemd

openssh is compiled with a patch to add a feature related to systemd

the system is using glibc (this is mandatory for systemd systems afaik anyway)

xz package was built using release tarballs published on GitHub and not auto-generated tarballs, the malicious code is missing in the git repository

So far, it seems openSUSE Tumbleweed, Fedora 40 and 41 and Debian sid were affected and vulnerable. Nobody knows what the vulnerability is doing exactly yet, when security researchers get their hands on it, we will know more.

OpenBSD, FreeBSD, NixOS and Qubes OS (dom0 + official templates) are unaffected. I didn't check for other but Alpine and Guix shouldn't be vulnerable either.

Gentoo security advisory (unaffected)

What lessons could we learn?

This is really unfortunate that a piece of software as important and harmless in appareance got compromised. This made me think about how could we protect the most against this kind of issues, I came to the conclusion:

packages should be built from source code repository instead of tarballs whenever possible (sometimes tarballs contain vendoring code which would be cumbersome to pull otherwise), at least we would know what to expect

public network services that should be only used by known users (like openssh, imap server in small companies etc..) should be run behind a VPN

OpenBSD style to have a base system developed as a whole by a single team is great, such kind of vulnerability is barely possible to happen (on base system only, ports aren't audited)

whenever possible, separate each network service within their own operating system instance (using hardware machines, virtual machines or even containers)

avoid daemons running as root as possible

use opensnitch on workstations (linux only)

control outgoing traffic whenever you can afford to

I don't have much opinion about what could be done to protect supply chain. As a packager, it's not possible to audit code of each software we update. My take on this is we have to deal with it, xz may certainly not be the only one vulnerable library running in production.

However, the risks could be reduced by:

using less programs

using less complex programs

compiling programs with less options to pull in less dependencies (FreeBSD and Gentoo both provide this feature and it's great)

Conclusion

I actually have two systems that were running the vulnerable libs on openSUSE MicroOS which updates very aggressively (daily update + daily reboot). There are no magic balance between "update as soon as possible" and "wait for some people to take the risks first".

I'm going to rework my infrastructure and expose the bare minimum to the Internet, and use a VPN for all my services that are for known users. The peace of mind will obtained be far greater than the burden of setting up WireGuard VPNs.