gemini://going-flying.com/~mernisse/24.gmi

In the previous three DNS posts I covered DNS from a fairly high level and then focused in on some key elements in BIND, the de-facto DNS server package. DNS is such an important part of inter-networking that it is bound to be a bit complex and given that complexity leads towards errors when humans are involved it is a pretty solid use-case for automation.

I mainly use Puppet to automate the state of my systems. I find it to be much more powerful than other tools and it allows me to specify a desired state instead of a set of tasks. It is also one of the most mature automation tools out there and is well supported on the platforms I use.

In general I automate three key pieces of my DNS infrastructure (other than making sure the required packages are installed and up-to-date, of course). 1) Zonefile generation 2) TLSA and DMARC records 3) DNSSEC key rotation and signing

Zonefile generation is easy, they are all generated from templates and concatenated together. This way I can specify records that may change in YAML using Hiera (Puppet's hierarchical key/value store), or resolve them via custom functions (as I do with the TLSA records for example) and have static blocks that are the same for all of my domains (like the NS and MX record sections).

My Puppet master runs various jobs that generate and rotate cryptographic keys. I wrote some custom plugins for LetsEncrypt certbot that lets Puppet manage the certificates and keys. Since they are available to Puppet I can generate the TLSA records that are used for DANE (a method to assure a client that the certificates they are getting from a TLS server are the correct ones). I provide the fingerprints used by the keys in the format the Apache (my webserver of choice) and BIND want using Facter (the Puppet host fact engine). The following Ruby code provides the custom facts to Facter.

# Generate the fingerprint of the SSL cert for things like # Certificate pinning and TLSA records. require 'fileutils' require 'openssl' Facter.add("ssl_pubkey_fingerprint") do confine :kernel => "Linux" setcode do cert = "/etc/ssl/certs/ssl.ub3rgeek.net_cert.pem" fingerprints = {} if File.exist?(cert) fd = File.read(cert) s_cert = OpenSSL::X509::Certificate.new(fd) p_key = s_cert.public_key().to_der() dgst = OpenSSL::Digest::SHA256.new(p_key) fingerprints['ssl.ub3rgeek.net'] = dgst.base64digest() end cert = "/etc/ssl/certs/going-flying.com_cert.pem" if File.exist?(cert) fd = File.read(cert) s_cert = OpenSSL::X509::Certificate.new(fd) p_key = s_cert.public_key().to_der() dgst = OpenSSL::Digest::SHA256.new(p_key) fingerprints['www.going-flying.com'] = dgst.base64digest() end fingerprints end end Facter.add("ssl_pubkey_hexdigest") do confine :kernel => "Linux" setcode do cert = "/etc/ssl/certs/ssl.ub3rgeek.net_cert.pem" digests = {} if File.exist?(cert) fd = File.read(cert) s_cert = OpenSSL::X509::Certificate.new(fd) p_key = s_cert.public_key().to_der() dgst = OpenSSL::Digest::SHA256.new(p_key) digests['ssl.ub3rgeek.net'] = dgst.to_s() end cert = "/etc/ssl/certs/going-flying.com_cert.pem" if File.exist?(cert) fd = File.read(cert) s_cert = OpenSSL::X509::Certificate.new(fd) p_key = s_cert.public_key().to_der() dgst = OpenSSL::Digest::SHA256.new(p_key) digests['www.going-flying.com'] = dgst.to_s() end digests end end

Finally, the most complex automated task is DNSSEC. DNSSEC is different from DNS over TLS (or DNS over HTTPS). DNSSEC does not seek to provide any privacy, rather it provides a chain of trust back to the root of the global DNS system so your resolver can verify that the answer it is getting is in fact the correct answer. DNSSEC is a huge topic but in general it allows zone administrators to cryptographically sign the records in their zones and any zones that they delegate to, so in going-flying.com you can trace the signatures from going-flying, to com to '.' (the root of DNS) and know for sure that someone isn't spoofing the answer.

I have a script that is called by puppet any time the DNS configuration changes (which is at least monthly due to various cryptographic key rotations and things like LetsEncrypt).

#!/bin/sh # dnssec-signzones (c) 2017-2018 Matthew J. Ernisse <matt@going-flying.com> # # Manage DNSSEC for zones, including rotating keys as needed. set -e PATH=$PATH:/sbin:/usr/sbin # Paths to BIND files KEYDIR="/etc/bind/keys" STATEDIR="/var/tmp" ZONEDIR="/var/cache/bind" # Timeouts DEACTIVATE_TIMEOUT="+3d" DELETE_TIMEOUT="+5d" PREPUBLISH_INTERVAL="+2d" REKEY_AGE="30" # days # delete any keys that are past their delete-by delete_old_keys() { local dtime=0 local now=$(date +"%s") for file in $(find ${KEYDIR} -name \*.key); do if grep -q 'key-signing' "$file"; then continue fi dtime=$(sed -ne 's/^; Delete:\s*[0-9]*\s*($.*$)/\1/p' \ "$file") if [ -z "$dtime" ]; then continue fi dtime=$(date -d "${dtime}" +"%s") if [ "$dtime" -le "$now" ]; then echo "Deleted expired key: $file" rm -- "$file" rm -- "$(echo "$file" | sed -e 's/\.key$/.private/')" fi done } find_domains() { for domain in $(ls -1 ${KEYDIR} | \ sed -ne 's/^K$[-\._a-z0-9]*$\.\+.*/\1/p' | sort -u); do echo $domain done } # return the latest (currently used) DNSSEC keyfile get_last_key() { if [ -z $1 ]; then echo "usage: get_last_key domain" return 127 fi local domain fn m mtime=0 domain="$1" for file in $(find ${KEYDIR} -name K${domain}\*.key); do if grep -q 'key-signing' "$file"; then continue fi keyage=$(sed -ne \ 's/^; Activate:\s\{1,\}[0-9]\{1,\}\s\{1,\}($.*$)$/\1/p' \ "$file") keyage=$(date -d "$keyage" +%s) if [ -z "$keyage" ]; then echo "get_last_key() failed to read activation time" return 1 fi if [ "$keyage" -gt "$mtime" ]; then mtime="$keyage" fn="$file" fi done echo "$(basename $fn)" } need_new_key() { if [ -z $1 ]; then echo "usage: need_new_key domain" return 127 fi local domain last_key keyage now domain="$1" now=$(date +%s) last_key=$(get_last_key "$domain") keyage=$(sed -ne \ 's/^; Activate:\s\{1,\}[0-9]\{1,\}\s\{1,\}($.*$)$/\1/p' \ "$KEYDIR/$last_key") keyage=$(date -d "$keyage" +%s) if [ $(( $now - $keyage )) -ge $(( ${REKEY_AGE} * 86400 )) ]; then echo "key ${last_key} needs rotating" return 0 fi return 1 } rotate_keys() { if [ -z $1 ]; then echo "usage: rotate_keys domain" return 127 fi domain="$1" last_key=$(get_last_key "$domain") # Logically this should go the other way, generate a new and then # if successful expire the old but dnssec-keygen refuses to generate # a successor key if there is no expire time on the current key.. echo "Setting expire times on ${last_key}" if ! dnssec-settime -K "$KEYDIR" -I "$DEACTIVATE_TIMEOUT" \ -D "$DELETE_TIMEOUT" "$last_key"; then exit 1 fi echo "Generating new key..." if ! dnssec-keygen -K "$KEYDIR" -S "$last_key" \ -i "$PREPUBLISH_INTERVAL" -q; then exit 1 fi } sign_domain() { if [ -z $1 ]; then echo "usage: sign_domain domain" return 127 fi domain="$1" for view in "public" "private"; do zonefile="${view}-${domain}.hosts" if [ -f "$ZONEDIR/$zonefile" ]; then sign_zone "$domain" "$zonefile" fi done } sign_zone() { if [ -z "$1" ] || [ -z "$2" ]; then echo "usage: sign_zone domain zonefile" return 127 fi domain="$1" zonefile="$2" echo "signing $zonefile for $domain" dnssec-signzone -S -K "$KEYDIR" -o "$domain" \ "$ZONEDIR/$zonefile" } # update the zone's serial -- this is no longer handled by puppet. update_serial() { if [ -z $1 ]; then echo "usage: update_serial domain" return 127 fi local new_serial old_rev old_rev_stripped old_serial=0 local zonefile if [ -f "${STATEDIR}/${domain}.serial" ]; then old_serial=$(< "${STATEDIR}/${domain}.serial") old_rev=$(echo "$old_serial" | sed -ne \ 's/^[0-9]\{,8\}$[0-9]\{,2\}$$/\1/p') old_serial=$(echo "$old_serial" | sed -ne \ 's/^$[0-9]\{,8\}$.*/\1/p') # Bash will think we're in octal if we feed it a number # with a leading zero. old_rev_stripped=$(echo "$old_rev" | sed -ne \ 's/^0$[0-9]$/\1/p') if [ -n "$old_rev_stripped" ]; then old_rev="$old_rev_stripped" fi fi new_serial="$(date +%Y%m%d)" if [ "$new_serial" -eq "$old_serial" ]; then if [ $(( $old_rev + 1 )) -lt 10 ]; then new_serial="${new_serial}0$(( $old_rev + 1 ))" else new_serial="${new_serial}$(( $old_rev + 1 ))" fi else new_serial="${new_serial}00" fi echo "$new_serial" > "${STATEDIR}/${domain}.serial" for view in "public" "private"; do zonefile="${ZONEDIR}/${view}-${domain}.hosts" sed -e "s/##SERIAL_NUMBER##/${new_serial}/" \ "${zonefile}.puppet" > "$zonefile" done } for domain in $(find_domains); do if need_new_key "$domain"; then rotate_keys "$domain" fi delete_old_keys update_serial "$domain" sign_domain "$domain" done

The script is complex because it needs to do several things but the for loop at the bottom tells the story of what is happening. The primary job of the script is to manage the DNSSEC keys, rotating them and expiring old ones, so since this script is run every time Puppet every time any of the DNS zonefiles are changed the first thing we do is to check to see if we actually need to rotate the keys. Once we do that we simply expire any old keys, update the serial numbers on the zones and then sign them. I can't imagine this script working verbatim for anyone else, it assumes a lot of things that are probably only true in my setup, but it should get anyone looking to automate DNSSEC with BIND well on their way.

DNS is one of the most important services provided on the Internet. It is used every time you type a name instead of an IP address. It is used for service discovery and configuration, e-mail SPAM detection, malware prevention and is the core method CDNs use to steer you to their closest edge location to optimize delivery of everything from JPEGs to critical software patches. Being comfortable with it gives a systems / network administrator tremendous control over network traffic and performance, it is an extremely powerful tool in the security administrator's portfolio, being able to short circuit command and control server connections and prevent malicious payloads from being downloaded, and any support people as trouble in DNS land is the root cause of so many end-user problems. I highly recommend _DNS and BIND_, by Cricket Liu and Paul Albitz, published by O'Reilly Media as further reading.

[06/14/2021 @13:30]: Sysadmining: DNS Four, Automation