Colin Cogle's Blog

"Do It Better: SSH" by Colin Cogle

Published November 7, 2023.

Abstract

SSH is easy to secure, but that doesn't mean there isn't more work to be done.

Let's talk about SSH. It is the quintessential Swiss army knife for system administrators. It can browse and transfer remote files, forward ports, proxy and tunnel traffic, but it's most famous for its eponymous job of giving you a remote shell, securely (hence its name). As processors have gotten faster and cheaper, and encryption has become almost frictionless, it's long since replaced Telnet for configuring embedded devices. In fact, even Microsoft acquiesced and made SSH a first-class citizen starting with Windows Server 2019.

Microsoft Learn: OpenSSH

You'd think with SSH everywhere, it would be easy to configure, right? Well, you are right. In fact, the default configuration of OpenSSH is pretty secure, but I guarantee you're still not implementing it as well as you could.

Installing It

I'm going to assume that your operating system either includes or is supported by a recent version of OpenSSH. We're going to configure things to support only the most modern features, so if you're trying to connect with an ancient version of macOS or PuTTY for Windows Vista, you might get shown the cold shoulder. We're dealing with security. Upgrade your software.

The Client

Every major desktop operating system — even Windows — now includes the OpenSSH client (the part you use to connect to other SSH servers). Open up a terminal, PowerShell window, Command Prompt, or what-have-you and try typing in `ssh`. I bet something will happen.

The Server

As for the OpenSSH server (the thing you log into), how you get that depends on your operating system. For Windows 10, Windows Server 2019, and newer, the OpenSSH client is usually installed by default, while the OpenSSH server exists as a Feature On Demand. It's built into macOS but dormant until you go into Sharing Preferences and enable Remote Login. Meanwhile, on Linux, you should try installing `openssh-server` and see what happens.

See also: Microsoft Learn

See also: macOS User Guide

If you're using a network device or some other embedded device, check and see what's available. Enterprise-oriented networking equipment often includes an SSH server. Embedded devices typically use a smaller SSH daemon like Busybox or Dropbear, neither of which I'm going to cover in detail in this article. This article is focused more on permanent installations, such as on a remote server.

Once you've installed an SSH server, you will have to edit text files to get it configured just the way we like it. UNIX and UNIX-like systems will install the configuration files into `/etc/ssh`. (Windows users, it's somewhere under `C:\ProgramData`.)

You could use a reference configuration like Mozilla's "Modern" OpenSSH setup,

or work with me while I create something newer. Open up the server's main configuration file, `sshd_config`. Most of the work we'll be doing will take place in this single file. Leave it open and keep reading.

Lock The Door, or Remove The Door

Let me state the obvious. Just because SSH is secure, why risk it? You are enabling and exposing a remote access method. That's the whole point, but it bears repeating. Consider how much risk you're willing to take, and don't push a half-assed solution into production.

The first thing to consider is where you will be accessing this remote server *from*. Just because we *can* open port 22 to the whole world doesn't mean that we should. Ask yourself:

I'll only be accessing this server from within my home, office, colo, etc.

Well, this makes it easy. Don't expose SSH to the public Internet. Do not add a firewall rule. Do not create a NAT policy. Do not pass Go and do not collect $200. Look, you're already done with this step.

But I could be connecting from anywhere!

This is the other extreme. Maybe you travel a lot. Maybe you're often on random Wi-Fi networks. Maybe your ISP only offers dynamic IP addresses. Or, you might be lazy, and that's fine, too.

In this case, just configure your firewall to allow SSH connections from anywhere. For example, in this made-up firewall pseudocode, we might make the following changes:

Every network should have some kind of firewall or network control device. Maybe it's a fancy SonicWall or Cisco device. Perhaps it's a simple Azure or AWS security group. Perhaps it's just the modem/router hybrid your ISP gave you. This part is left as an exercise to the reader. (I believe in you.)

Okay, fine. I suppose I only need access from a select few locations.

This scenario is often used in businesses, and mine is no exception. My company policy is to make remote management interfaces only available to a select few IPv4 addresses and IPv6 subnets. This does mean that remote employees (myself included) need to use a VPN or a remote desktop, but that's a small price to pay for a surprisingly-effective security implementation. As I once explained this, a hacker can't knock on the door if they can't get to it in the first place.

In this scenario, you will expose your SSH server to the Internet, setting up the necessary firewall rules (and IPv4 NAT policies, if you need that older protocol). However, in this case, configure your firewall rules so that access is only allowed from certain IP addresses or subnets. For example:

If your remote-side ISP only provides you dynamic IP addresses, maybe you can still configure this scenario. Does your remote firewall offer some kind of VPN or VPN-like feature? If so, set that up, then configure your firewall to allow SSH connections from VPN clients instead of the entire Internet.

Security Through Obscurity?

Some people will tell you that you should never run SSH on the default port of TCP 22. They'll advise you to pick another number. While that may defer script kiddies and automated scans, it's trivial to find an SSH service. For example, I could just scan all of your ports in the span of a few minutes:

Or, I can just type your IPv4 address into Shodan and get a report instantly.

In either case, I found that your SSH server is running on port 42069, let's say. I can also see what version of OpenSSH it uses, and begin to see how it might be configured. Don't waste your time with security theater and just leave it on port 22.

One Lock, One Key

SSH relies on public key cryptography to authenticate servers and users. There are four major key types still in use today:

RSA is the tried and true key generation algorithm. It's the oldest, but widely supported. It does require large keys, which results in slower cryptography.

DSA was another popular algorithm, but its time has passed. It's been removed from OpenSSH 7.0 and newer, finally.

ECDSA was the first major challenger to the RSA hegemony. It uses smaller keys without compromising security. However, it's imperfect, and some people believe that the United States government may have backdoored it. (If so, Edward Snowden hasn't told us about it.)

EdDSA is the newest algorithm that fixes the weaknesses of ECDSA and matches or exceeds the security of RSA. It's also much faster. One specific curve, Curve25519, has taken the cryptography world by storm and is gaining wide support in the industry.

My recommendation is to use EdDSA for everything, falling back to ECDSA or RSA if needed. However, since we're only targeting the newest versions of OpenSSH, let's forget that anything besides Curve25519 exists.

Host Keys

When you start SSH for the first time, you'll notice that it generates many different kinds of host keys: RSA, ECDSA, and Ed25519 (and DSA if you have an older version). There's no need to use anything but the newest, so let's disable and delete the others.

Stop your SSH server, and edit your `sshd_config` file. Comment out all of the other `HostKey` lines.

Now, delete all other host keys and restart OpenSSH:

Now, only clients that support Ed25519 host keys can even attempt to log in. This step by itself will break many old IP scanners, botnets, and script kiddies' tools. (I look through my logs sometimes and see a lot of connections failing because `ssh-rsa` is unsupported.)

Keep Only The Best Key Exchanges

Now that we have only the best host key type remaining, let's take a look at the next stage of the SSH connection, when the client and server exchange keys and prepare to set up an encrypted session. Just like with host keys, there are a lot of key exchanges supported. We should only support the most secure ones.

And, just like with host keys, older versions of OpenSSH may not be able to connect. But if all our computers are using, let's say, OpenSSH 8, why bother supporting older versions?

You can run the `ssh -Q kex` command to see all key exchanges supported by your version of OpenSSH, but there are three that we'll be using:

`sntrup761x25519-sha512@openssh.com` provides excellent post-quantum security that doesn't affect performance too badly. We'll use that first.

`curve25519-sha256@libssh.com` and `curve25519-sha256` provide excellent security and performance. Enable them both. (I'm not sure what the difference is between them — toot at me if you do.)

(Complaints to @colincogle@mastodon.social)

We're going to leave these off the list for varying reasons:

The `ecdh` curves use ECDSA, and we prefer EdDSA. They're fine to enable if needed.

`diffie-hellman-group-exchange-sha256` is older and slower, but still secure. Still, I won't be using this.

Anything like `diffie-hellman-groupN` should be avoided.

Go ahead and put the algorithms you want into the `sshd_config` file, like so:

With all of these multiple-choice directives, an SSH client and server will go down the list, from left to right, and use the first thing that they mutually agree on. Always arrange them from most secure to least secure, and omit ones you don't want to support.

Disabling Less-Secure Diffie-Hellman Moduli

Though we've disabled all Diffie-Hellman key exchanges in this topic, you may need to re-enable them if you need to support an older client. Even if you don't, it's still good practice, just in case they get re-enabled for whatever reason. There is a file, `/etc/ssh/moduli`, which needs to be edited. While it can be edited by hand, the fine folks at Mozilla have given us a Bash one-liner:

(Source: "Mozilla Security Guidelines: OpenSSH")

Now, any DH key exchanges will use 3072-bit RSA at a minimum.

Ciphers for Bulk Encryption

Now that both sides can exchange keys securely, we need to look at ciphers. We have far fewer options here, but that's fine because what's offered is quite secure. You can use `ssh -Q cipher` to see what's supported.

The `aes256-gcm@openssh.com` and `aes128-gcm@openssh.com` ciphers are the most secure, and AES in Galois Counter Mode is hardware-accelerated on most processors. These are what most everyone should use.

`chacha20-poly1305@openssh.com` is another secure 256-bit algorithm that may be faster on some embedded devices and older ARM processors.

`aes256-ctr`, `aes192-ctr`, and `aes128-ctr` are older but still secure implementations of AES. They might not be as fast as the AES-GCM ones.

Do not use any of these:

CBC cipher modes are subject to padding oracle attacks.

CAST-128, 3DES, and Blowfish use 64-bit blocks which are considered too small these days.

SHA-1 collisions can be manufactured.

Rijndael is a pre-standards version of AES.

Let's add this line to our `sshd_config` file:

Performance Considerations

While both provide security that won't be broken anytime soon, I would recommend using AES-256 over AES-128 in most cases. However, I sometimes run graphical apps over SSH by using Waypipe (and the older X forwarding), and I have noticed AES-128 provides a significant speed boost in that case. If you plan to use Waypipe, X forwarding, or you plan to transfer a lot of files via SSH, you may want to consider listing AES-128 first.

My servers prefer AES-256, but my desktop prefers AES-128. If you're unsure, you can also set per-host settings on the client side — more on that later.

Hashing and Data Verification

Finally, there are MACs — Message Authentication Codes — used to verify that encrypted data has not been modified, intentionally or otherwise, in transit. Again, you can use `ssh -Q MACs` to view what's supported, and we'll promptly cull this long list.

The SHA-2 family is the most secure, and there isn't much of a performance impact on modern hardware. Sort by number of bits, in descending order.

Avoid or disable anything with the following characteristics:

1. Omit anything using 64-bit or 96-bit digests, as those are too small for modern use.

2. Omit anything using broken algorithms like MD5 or SHA-1.

3. Avoid anything using weak algorithms like RIPEMD-160.

4. Avoid anything *not* using ETM (Encrypt-then-MAC mode), as the alternative of hashing before encrypting can lead to weaknesses.

That doesn't leave us with much, but it leaves us with good options. Add this line to your `sshd_config`.

Other SSHd Options

Now, we're going to explore the other lines in the `sshd_config` file. Many of these need little explanation, so I'm going to go through this next section quickly.

Always Use SSH Version 2

SSH version 1 was deprecated back in 2006 with RFC 4251,

and all support was removed from OpenSSH 7.6 back in 2017. It's dead. Get on with your life.

If it makes you feel better, try adding this to your `sshd_config` and hope it doesn't complain about an unsupported option:

Don't Use Passwords, Use Keys

Take it from Mister IT Guy over here: **don't use passwords if there's a better option**. Generating SSH keys is a quick and painless procedure. Disable password-based logins and stop credential stuffing before it occurs.

If your server and clients are joined to a Kerberos realm or an Active Directory domain, you can look into setting up OpenSSH's native Kerberos and GSSAPI authentication mechanisms. A Kerberos ticket is arguably even more secure than a stealable key. I haven't worked with OpenSSH and Kerberos in many years, so I won't pretend to know how to configure it.

Get Lost, Root!

There is literally no good reason to be working as `root` on a daily basis. Anyone sane should be using `sudo` to elevate their own permissions. However, it's possible you set a root password long ago, or you may have accidentally associated a public key with the root account.

Regardless, logging in as `root` over a network is a pretty terrible idea. Unless you can literally sit down and explain why you should not make this change, you should make this change:

Do You Really Need X Forwarding?

X forwarding is a great way to run graphical Linux apps over an SSH connection, and have them appear on your remote screen like a native, local app.

However, many Linux distributions have switched Xorg for Wayland. If you use Wayland, you should be using Waypipe to run graphical apps over SSH, as I found it much faster. Thus, if you're using Wayland, your server is headless, or all of this just plain sounds like a dumb idea, comment out the `X11Forwarding` line in `sshd_config`, or set it to `no`:

You can comment out any other options that start with `X11` if you like a nice, clean configuration file; however, they won't have any effect.

More Logging

If something goes wrong, you're going to want to refer to your logs, so let's make sure we generate some useful data. If you'd like to know which keys are being used to log into your server, bump up the logging level.

You can also log what people do via SFTP file transfers.

After a successful login, you should show the user their previous login to make sure it matches what they expect.

Storing SSH Fingerprints in DNS

When you connect to a new SSH server for the first time, you will be asked to verify the SSH fingerprint, so that you can avoid a future man-in-the-middle attack. Most people will blindly accept it. However, this process can be automated by putting the server's host key fingerprint(s) into DNS.

Pick a public name for your server, and then feed it to the `ssh-keygen -r` command, like so:

If you get more than two records, then you must have forgotten to delete your unwanted SSH host keys in the previous steps. That's fine. We only want the Ed25519 host key fingerprints, which all begin with the number 4. Ignore the others.

Now, go to wherever you go to edit your DNS zone. This might be Cloudflare, a local BIND server, or your registrar.

If you're unlucky enough to be using Microsoft DNS Server, I have you covered, believe it or not.

Next, add two new SSHFP records, one for each line. If you're unluckier still and your DNS host doesn't support SSHFP records (looking at you, GoDaddy and Network Solutions), then you're out of luck.

Note that this is all for naught if you don't use DNSSEC. If your zones aren't signed, please look at a calendar, see what year it is, then ask yourself why you haven't secured your infrastructure yet. Even GoDaddy realized how important secure DNS is to the Internet ecosystem and stopped charging extra for DNSSEC.

(Even those scummy capitalists can do the right thing once in a while.)

Now, edit your *local* machine's `/etc/ssh/ssh_config` file and add the following two lines:

Now, when you connect to an SSH server for the first time, SSHFP records will be retrieved from DNS, and your computer will help you decide if you're connecting to the right server.

Bonus: PowerShell Remoting over SSH

Believe it or not, this is now supported! If using SSH as a native transport for PowerShell remoting sounds like something you want, add this line, substituting the correct path for your particular system:

(Source: Microsoft Learn)

Setting Up Public-Key Authentication

Now that we've set up OpenSSH, we're ready to expose it to the Internet. However, there is one problem we haven't addressed: we can't log in. Remember, we disabled password-based logins earlier! To make SSH usable and secure, we're going to log in with a keypair.

Generating Your First Keypair

If you already have one you'd like to use, skip this entire section. On your computer (not the server), generate a new key. Open up a terminal and, like before, we're going to use Ed25519.

You'll be prompted to create a password to protect your key against theft. Pick something strong, or throw your opsec to the wind and use no password at all.

Now, why did I pick the name `id_ed25519`? Keys named like `id_<algorithm>` will be tried automatically when connecting to servers. If you use a different name, you'll need to specify an `IdentityFile` manually for each remote server you connect to.

In the `.ssh` folder in your home folder, you will have two files. `id_ed25519` is your private key. That should never leave your computer (except when being securely backed up), and never fall into the hands of an attacker. This goes double if you decided not to protect your key with a password. The other file is `id_ed25519.pub`, which you can and will share with the world.

Some people have strong philosophical opinions about using one keypair per server. I disagree. I believe a key is more closely associated with a person (or a person's device) and I have no problem using the same keypair for many servers. However, I do create separate ones for personal use and work use.

Getting Your Public Key to the Server

Now that you have a public key, we need to get it to the server. Copying and pasting is usually the simplest way, so open up `id_ed25519.pub` in a text editor and copy that whole thing.

On the server, navigate to `~/.ssh` and edit the `authorized_keys` file. Paste that public key onto a blank line, then hit save!

Moment of Truth

Let's give it a shot!

If your local and remote usernames are the same, you can leave off the `user@`.

If you've set up SSHFP records, you should see "Verified host key found in DNS." Accept the correct fingerprint and you should be logged in!

Multi-Factor Authentication

By this point, we've set up a pretty secure SSH server, but I'm sure my astute and security-conscious readers are getting ready to send me angry toots about me teaching them to expose a remote access method to the Internet with only single-factor authentication enabled. If you're using Kerberos or only allowing certain IP addresses to connect, this may not be necessary. Plus, keys are far less likely to be stolen than passwords. Still, let me show you how to implement MFA via the classic TOTP method.

These steps only apply to Linux systems that use PAM for user authentication, which should be almost all of them these days. On your server, find and install the Google Authenticator PAM module. On Debian-like systems, a simple `apt install libpam-google-authenticator` will do the trick.

(Or grab `google-authenticator-libpam on GitHub.)

Since we're about to mess with the thing that authenticates users, open up another terminal or SSH session on the server, and *keep it open until we're done and we've tested that it works*. Otherwise, a typo could lock you out and give you a very hard problem to fix.

Edit the file `/etc/pam.d/sshd` and add the following line to the top, above the one that ends in `pam_nologin.so`:

If you've made other changes to your PAM configuration files, then you may need to do some debugging.

Add ` debug` to the end of that `pam_google_authenticator.so` line to generate more debugging output in your logs.

Next, edit your `/etc/ssh/sshd_config` file, adding this new stuff and making sure these other lines are present, too:

Now, on the server, run the `google-authenticator` command to generate your secret (as a QR code) and some backup codes. Scan the QR code with your authenticator app (despite the name, it doesn't need to be Google Authenticator) and be sure to save the backup MFA codes somewhere safe.

Moment of truth, again! In a new window, try to SSH into your server again. If all goes well, you'll be prompted for an MFA code before your SSH login completes.

Note that all SSH users will be required to use MFA. If they haven't set up MFA, then PAM will deny their SSH login every time. To work around this, remove MFA enforcement by adding `nullok` to that PAM.d line we added earlier.

Client-Specific Settings

Now that we've configured our servers, we can create profiles for our connections so that we don't need to remember and type in every single detail every time. OpenSSH gives us two files: the system-level `/etc/ssh/ssh_config` and the user-specific `~/.ssh/config`. Both files have the same syntax. Whichever one you choose to edit depends on your personal preferences.

Note that, when you're writing this file, you may be connecting to SSH servers that you do not control. You may want to leave all secure options in here, and let your SSH servers be more restrictive.

In Conclusion

SSH isn't hard. It's actually quite simple to configure and use, certainly more so than a modern email server. However, we've taken the time to adjust some default settings, break some compatibility, and we've hardened things significantly. You can now SSH safely, and that's all that matters.