Layered VPNs

I went through a few configurations recently trying to get the perfect networking setup.

I don't think my requirements are too out of the question:

Have one domain per machine

Which works locally and remotely

If it's local I don't want the packets leaving the home network

mDNS

Link

This is a highly probable solution for most standard setups, many distros come with an avahi daemon already configured, and even android can by default resolve them. mDNS is an interesting technology which lets a client resolve an address by asking other devices who claims that address. As the name implies these exchanges are done by multicast.

Service Discovery

mDNS is used to many many genius purposes you might not even realize. If you run `avahi-browse -at` then you can see a list of domains that different machines on your network advertise. These take the form of services for example `_scanner._tcp` or `_guix_publish._tcp`. So now when your phone wants to cast a video to a tv it asks for the domain `_googlecast._tcp` and then lets you choose which one of the devices advertising that domain you want to cast to as multiple devices are often offer one service and thus advertise the same domain. This is used for all kinds of service discovery.

.local domains

The same mechanism can be used to locally resolve another type of domain, with the TLD .local by adding mdns to your `/etc/nsswitch.conf` file. Afterwards whenever I ask to go to `hydra.local` for example, since my main machine has the hostname `hydra` then it goes through the mDNS song and dance and just connects to that machine. It feels like magic the first time you play around with it.

What could possibly go wrong

mDNS is a bit prone to confusing itself with bizarre networking setups, but that's definitely not anyone's but my fault. There's one more thing about mDNS, it's awesome at working locally, however to make it work over longer distances you need to actually make sure the multicast packets travel that distanace. That is not at all difficult, you can set up a so called reflector which bridges mDNS packets from one network to another, however I found a perhaps even nicer method.

WireGuard

Link

WireGuard is an awesome project which is already part of the Linux Kernel since version 5.6. It allows you to create a device that acts as an overlay network over your other devices. It uses a Peer to Peer setup, where every machine has a few bits of information.

Addresses

Private Key

AllowedIPs

Endpoint (optional)

The Private Key identifies the device to other peers that add it. The Endpoint tells wireguard how to reach that peer through the real network, and it keeps those packets end to end encrypted. AllowedIPs works both as a filter for incoming traffic, and as a routing table entry. You don't actually have to include the entry of the external endpoint IP, this is mainly to include other machines that can be routed trustfully through that peer. For example, if I have a subnet of 2001::/64 connected to Peer A, then the other peers will add 2001::/64 to their AllowedIPs list, which for one allows those machines to send packets to you through the peer you add that to, but it also adds a route so that when you send packets to that subnet, they will get sent to that peer who knows what to do with them. Thus you can use wireguard to bridge any two local LANs with just two devices that can see each other, or even route all traffic through the wireguard network by adding :: to the AllowedIPs list. Which also means that you can use any VPS provider to make a public peer your devices connect to, and then they can see each other through that central peer. You just add the internal subnet as the AllowedIP of the central peer, and then add each device and its designated IP to the central server.

Wireguard however is quite rigid, it will only use one endpoint no matter the context, unless you setup a dynamic DNS and tell it to resolve over that, but that starts to complicate what should be quite a simple goal. There are many cool projects that expand and build on the wireguard base, it is good that wireguard remains simple, it's a part of the kernel and has many many eyes looking at the security of it, in which case simplicity helps a great deal. As such, again, there is still a project that can surpass this at least in terms of dynamicity but we will return to wireguard.

Yggdrasil

Link

The main ingredient is a magnificent project named Yggdrasil, which is an experimental (... more like research as far as I understand it, the thing is very stable, but it's basically unique in what it does and is being tested) routing project of sorts. Yggdrasil is very straightforward to setup, you run the thing and you are done. Wait what???, yes really. When you run the daemon, you suddenly receive an IPv6 address that is routable on the local network. You can now reach any other yggdrasil device that you can reach with mDNS, since that is how yggdrasil discovers its peers, the kicker here is you can also reach any peer that is connected to your peers. Yggdrasil is a routing project after all and so it uses a distributed hash map across any network of yggdrasil devices to locate and route its packets, and very fast at that. You can also add a peer explicitly to connect to any yggdrasil device even outside of mDNS range, which of course, connects you to every device reachable from that node. Thus by connecting to a public peer you have the entire yggdrasil network at your fingertips, including any device you connect to it. There are even some webpages accessible only through yggdrasil.

ff-radio

And it optimizes the routes well, so when you're at home and your devices peer directly, then it sees that the fastest route is direct, even when your connection to the clearnet goes down (which happens not rarely here), yggdrasil still finds routes right at home. If you want to make the address persist, you switch off yggdrasil's autoconfiguration and set an explicit private key, which is used to generate the address and to E2E encrypt all traffic going through yggdrasil similarly to wireguard.

ZeroTier

Link

ZeroTier is an enterprise VPN done exquisitely. Their client is open source, and available in many repos, you install it, run `zerotier-cli join <network-id>`, allow it in the management console and you're done. The nodes in the network see each other and based on your configuration, they can get auto-assigned both IPv6 and IPv4 addresses, and you can turn on and off multicasts and broadcasts accross that network. All around a great experience. But the traffic will still go through their servers no matter what if I'm not mistaken. Otherwise however it is an autoconfiguration lover's dream, even has some DNS features I didn't even get to yet. And apparently you can even self-host it?

The final setup

What I ended up with is a combination of the above. I let yggdrasil do the heavy lifting, that address connects over whatever is best at the moment, either direct peering at home, over an explicit public yggdrasil peer or a peering discovered over zerotier. The latter two form a redundancy as the closest zerotier server I could find is in switzerland, and it's a waste to route traffic through there when there are yggdrasil peers here in the czech republic. However those peers sometimes go down and/or change addresses and I want to maintain a connection, and be able to correct that address over the network. The little paranoid sprinkle on top is that WireGuard has shown to be extremely fast, so it doesn't bring much latency at all to the table and it's just a bit of extra peace of mind that I am protected by something inspected rigorously enough to get past Linus. So in the end I have an explicit hostfile entry pointing to a wireguard address, the endpoint of which goes to the yggdrasil address of that device and yggdrasil takes it from there routing as it sees fit.

It's been absolutely lovely.

Quick tip for IPv4 over IPv6

I like IPv6 my entire internal setup is absolutely free of any v4. But I can't really go reconfiguring much on my friends PC, he was interested enough in Yggdrasil to try it out with me and we connected and realised that our game didn't like IPv6 and yggdrasil only supports IPv6 (good). Some games don't like that or simply have the IP input form reject it, such as Worms: Armageddon. And of course there are many many different ways to bridge IPv4 over IPv6 but most of them are way more difficult than this. You can actually just use an ssh tunnel.

An SSH tunnel takes a port on one machine and connects any connection to it as if it was a local connection to a port on a completely different machine.

So for example:

will allow you to tell Worms to connect to 127.0.0.1, which gets caught by the tunnel and goes over the network to whatever thing you've connected to and locally goes right into the server there on that other side. Trivial, I love it.