Configuration deployment made easy with drist

Author: Solène

Date: 29 November 2018

Tags: unix drist automation

Comment on Mastodon

Hello, in this article I will present you my deployement tool **drist** (if you

speak Russian, I am already aware of what you think). It reached a feature

complete status today and now I can write about it.

As a system administrator, I started using *salt* a few years ago. And

honestly, I can not cope with it anymore. It is slow, it can get very

complicated for some tasks like correctly ordering commands and a

configuration file can become a nightmare when you start using condition in it.

History

I also tried alternatives like *ansible*, *puppet*, *Rex* etc... One day, when

lurking in the ports tree, I found **sysutils/radmind** which got a lot

interest from me even if it is really poorly documented. It is a project from

1995 if I remember correctly, but I liked the base idea. *Radmind* works with

files, you create a known working set of files for your system, and you can

propagate that whole set to other machines, or see differences between the

reference and the current system. Sets could be negative, meaning that the

listed files should not be present on the system, but it was also possible to

add extra sets for specific hosts. The whole thing is really really cumbersome,

this requires a lot of work, I found little documentation etc... so I did not

used it but, that lead me to write my own deployment tool using ideas from

*radmind* (working with files) and from *Rex* (using a script for doing

changes).

Concept

**drist** aims at being simple to understand and pluggable with standard tools.

There is no special syntax to learn, no daemon to run, no agent, and it relies

on base tools like awk, sed, ssh and rsync.

**drist** is cross platform as it has a few requirements but it is not well

suited for deploying on too much differents operating systems.

When executed, **drist** will execute six steps in a specific order, you can

use only steps you need.

Shamelessly copied from the man page, explanations after:

1. If folder **files** exists, its content is copied to server rsync(1).

2. If folder **files-HOSTNAME** exists, its content is copied to server using rsync(1).

3. If folder **absent** exists, filenames in it are deleted on server.

4. If folder **absent-HOSTNAME** exists, filenames in it are deleted on server.

5. If file **script** exists, it is copied to server and executed there.

6. If file **script-HOSTNAME** exists, it is copied to server and executed there.

In the previous list, all the existences checks are done from the current

working directory where drist is started. The text **HOSTNAME** is replaced by

the output of `uname -n` of the remote server, and files are copied starting from

the root directory.

drist does not do anything more. In a more litteral manner, it copies files to

the remote server, using a local filesystem tree (folder **files**). It will

delete on the remote server all files present in the local filesystem tree

(folder **absent**), and it will run on the remote server a script named

**script**.

Each of theses can be customized per-host by adding a "-HOSTNAME" suffix to the

folder or file name, because experience taught me that some hosts does require

specific configuration.

If a folder or a file does not exist, **drist** will skip it. So it is possible

to only copy files, or only execute a script, or delete files and execute a

script after.

Drist usage

The usage is pretty simple. **drist** has 3 flags which are optionals.

- -n flag will show what happens (simuation mode)

- -s flag tells drist to use sudo on the remote host

- -e flag with a parameter will tell drist to use a specific path for the sudo

program

The remote server address (ssh format like user@host) is mandatory.

$ drist my_user@my_remote_host

drist will look at files and folders in the current directory when executed,

this allow to organize as you want using your filesystem and a revision control

system.

Simple examples

Here are two examples to illustrate its usage. The examples are easy, for

learning purpose.

Deploying ssh keys

I want to easily copy my users ssh keys to a remote server.

$ mkdir drist_deploy_ssh_keys

$ cd drist_deploy_ssh_keys

$ mkdir -p files/home/my_user1/.ssh

$ mkdir -p files/home/my_user2/.ssh

$ cp -fr /path/to/key1/id_rsa files/home/my_user1/.ssh/

$ cp -fr /path/to/key2/id_rsa files/home/my_user2/.ssh/

$ drist user@remote-host

Copying files from folder "files":

/home/my_user1/.ssh/id_rsa

/home/my_user2/.ssh/id_rsa

Deploying authorized_keys file

We can easily create the authorized_key file by using cat.

$ mkdir drist_deploy_ssh_authorized

$ cd drist_deploy_ssh_authorized

$ mkdir -p files/home/user/.ssh/

$ cat /path/to/user/keys/*.pub > files/home/user/.ssh/authorized_keys

$ drist user@remote-host

Copying files from folder "files":

/home/user/.ssh/authorized_keys

This can be automated using a makefile running the cat command and then running

drist.

all:

cat /path/to/keys/*.pub > files/home/user.ssh/authorized_keys

drist user@remote-host

Installing nginx on FreeBSD

This module (aka a folder which contain material for drist) will install nginx

on FreeBSD and start it.

$ mkdir deploy_nginx

$ cd deploy_nginx

$ cat >script <<EOF

test -f /usr/local/bin/nginx

if [ $? -ne 0 ]; then

pkg install -y nginx

fi

sysrc nginx_enable=yes

service nginx restart

EOF

$ drist user@remote-host

Executing file "script":

Updating FreeBSD repository catalogue...

FreeBSD repository is up to date.

All repositories are up to date.

The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:

nginx: 1.14.1,2

Number of packages to be installed: 1

The process will require 1 MiB more space.

421 KiB to be downloaded.

[1/1] Fetching nginx-1.14.1,2.txz: 100% 421 KiB 430.7kB/s 00:01

Checking integrity... done (0 conflicting)

[1/1] Installing nginx-1.14.1,2...

===> Creating groups.

Using existing group 'www'.

===> Creating users

Using existing user 'www'.

[1/1] Extracting nginx-1.14.1,2: 100%

Message from nginx-1.14.1,2:

===================================================================

Recent version of the NGINX introduces dynamic modules support. In

FreeBSD ports tree this feature was enabled by default with the DSO

knob. Several vendor's and third-party modules have been converted

to dynamic modules. Unset the DSO knob builds an NGINX without

dynamic modules support.

To load a module at runtime, include the new `load_module'

directive in the main context, specifying the path to the shared

object file for the module, enclosed in quotation marks. When you

reload the configuration or restart NGINX, the module is loaded in.

It is possible to specify a path relative to the source directory,

or a full path, please see

https://www.nginx.com/blog/dynamic-modules-nginx-1-9-11/ and

http://nginx.org/en/docs/ngx_core_module.html#load_module for

details.

Default path for the NGINX dynamic modules is

/usr/local/libexec/nginx.

===================================================================

nginx_enable: -> yes

Performing sanity check on nginx configuration:

nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok

nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful

nginx not running? (check /var/run/nginx.pid).

Performing sanity check on nginx configuration:

nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok

nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful

Starting nginx.

More complex example

Now I will show more complexes examples, with host specific steps. I will not

display the output because the previous output were sufficient enough to give a

rough idea of what drist does.

Removing someone ssh access

We will reuse an existing module here, a user should not be able to login

anymore on its account on the servers using the ssh key.

$ cd ssh

$ mkdir -p absent/home/user/.ssh/

$ touch absent/home/user/.ssh/authorized_keys

$ drist user@server

Installing php on FreeBSD

The following module will install php and remove the opcache.ini file, and will

install php72-pdo_pgsql if it is run on server *production.domain.private*.

$ mkdir deploy_php && cd deploy_php

$ mkdir -p files/usr/local/etc

$ cp /some/correct/config.ini files/usr/local/etc/php.ini

$ cat > script <<EOF

test -f /usr/local/etc/php-fpm.conf || pkg install -f php-extensions

sysrc php_fpm_enable=yes

service php-fpm restart

test -f /usr/local/etc/php/opcache.ini || rm /usr/local/etc/php/opcache.ini

EOF

$ cat > script-production.domain.private <<EOF

test -f /usr/local/etc/php/pdo_pgsql.ini || pkg install -f php72-pdo_pgsql

service php-fpm restart

EOF

The monitoring machine

This one is unique and I would like to avoid applying its configuration against

another server (that happened to me once with salt and it was really really

bad). So I will just do all the job using the hostname specific cases.

$ mkdir my_unique_machine && cd my_unique_machine

$ mkdir -p files-unique-machine.private/usr/local/etc/{smokeping,munin}

$ cp /good/config files-unique-machine.private/usr/local/etc/smokeping/config

$ cp /correct/conf files-unique-machine.private/usr/local/etc/munin/munin.conf

$ cat > script-unique-machine.private <<EOF

pkg install -y smokeping munin-master munin-node

munin-configure --shell --suggest | sh

sysrc munin_node_enable=yes

sysrc smokeping_enable=yes

service munin-node restart

service smokeping restart

EOF

$ drist user@incorrect-host

$ drist user@unique-machine.private

Copying files from folder "files-unique-machine.private":

/usr/local/etc/smokeping/config

/usr/local/etc/munin/munin.conf

Executing file "script-unique-machine.private":

[...]

Nothing happened on the wrong system.

Be creative

Everything can be automated easily. I have some makefile in a lot of my drist

modules, because I just need to type "make" to run it correctly. Sometimes it

requires concatenating files before being run, sometimes I do not want to make

mistake or having to remember on which module apply on which server (if it's

specific), so the makefile does the job for me.

One of my drist module will look at all my SSL certificates from another

module, and make a reed-alert configuration file using awk and deploying it on

the monitoring server. All I do is typing "make" and enjoy my free time.

How to get it and install it

- Drist can be downloaded [at this address](ftp://ftp.bitreich.org/releases/drist/drist-v1.02.tgz).

- Sources can be cloned using `git clone git://bitreich.org/drist`

In the sources folder, type "make install" as root, that will copy drist binary

to /usr/bin/drist and its man page to /usr/share/man/man1/drist.1

For copying files, drist requires rsync on both local and remote hosts.

For running the script file, a sh compatible shell is required (csh is not working).