Capsule Log Retention

I wrote previously that I had setup stats based on my servers access logs. I noted at the bottom that I was clearing my logs to keep somewhat compliant with user privacy.

Deleting logs older than 7 days

I wrote a daily cronjob to remove any log lines older than 7 days from my access.log

#!/usr/bin/env bash

set -e

LOGFILE=$1

mindate=$(head -n1 $LOGFILE | cut -f1 | cut -d'T' -f1)
maxdate=$(date --date="-6 days" -u -Id)

echo "Deleting log lines from ${mindate} to ${maxdate}"

sed -i -E "/${mindate}/,/${maxdate}/d" $LOGFILE

echo "Cleared logs"

Date command

I had no idea you could do relative dates! I am not surprised bur it was nice to find out since it meant I could do this entirely in bash.

What this is doing

I essentially take two dates: the first date in the file, and the inclusive date I want to remove logs for. Then using the magic of sed, remove all the lines that match between those dates.

Sed

I do a lot of bash scripting for """productivity""". Sed is an amazing utility that I find allows me to do so much without needing to expand into other languages like awk or python.

Obfuscating any personal data

The only "personal data" my access logs have are IPs. I am kicking around a good way to so I can still debug and trace requests, but for now keeping a week of logs should be compliant and hopefully personally reassuring.

Conclusion

Honestly, I find myself writing small script files like this (though this is basically just a sed command) to do day-to-day tasks. For instance my 'todo' app for work, to track asks, essentially just echos the args into a unique file in a directory. Then you can print the file contents to see whats open. Then using their UID to mark as done.

So I was happy to be able to quickly write this up to clear old logs. Do you do log rotation? Or just delete the old files?

Gemlog

Home

-- Response ended

-- Page fetched on Fri Apr 26 08:08:43 2024