Part of the Raspberry Pi Reliability series.

When something does go wrong, having collected logs somewhere other than your Raspberry Pi’s SD card is very helpful. Depending on how your Pi is set up:

• Logs may only be kept for the current boot or may be stored only in RAM, so after power-cycling the Pi you’ll have no log of what happened.
• Or, if SSH isn’t working and you don’t have easy physical access to the Pi, you’re in the same situation: no logs.

I’ve been using Logz.io’s community/free plan at home for several years now; it’s more than adequate for this use case. The remainder of this post will focus on setting up rsyslog with Logz.io, but in any case your takeaway from this post should be “remote log collection is helpful.”

Sending system logs to Logz.io with rsyslog

You should peruse the Logz.io documentation (or the docs for your preferred logging service) to see all your log shipping options, but here’s the TL;DR on how I set this up on my Linux systems:

TMP_DIR=$(mktemp -d 2>/dev/null || mktemp -d -t 'logz-linux')
cd "$TMP_DIR"
curl -sLO https://github.com/logzio/logzio-shipper/raw/master/dist/logzio-rsyslog.tar.gz
tar xzf logzio-rsyslog.tar.gz
sudo rsyslog/install.sh -t linux -a "$MY_LOGZIO_TOKEN" -l "listener.logz.io"

That’s it!

Configuring rsyslog to minimize SD card wear

The default rsyslog setup is fine for anything that’s not running from a microSD card … so I customize it on the Raspberry Pi.

First, edit /etc/fstab and create a tmpfs for /var/spool/rsyslog. Feel free to change the 25M to something like 50M if your Pi has the memory headroom:

tmpfs  /var/spool/rsyslog  tmpfs  defaults,noatime,nosuid,nodev,noexec,size=25M  0  0

Then we need to tell rsyslog how much disk space it’s allowed to use. The rsyslog docs specifically warn that it doesn’t handle running out of disk space gracefully.

In /etc/rsyslog.d/22-logzio-linux.conf you’ll notice a line like $ActionQueueMaxDiskSpace 1g. That won’t do, since we just put it in a 25MB filesystem in RAM!

Change that line to make the limit just under the size of your new /var/spool/rsyslog filesystem:

$ActionQueueMaxDiskSpace 24m

Reboot the Pi when you’re done.

Watching for critical errors with Logz.io

Logz.io’s alerts feature is useful, and if you’re sending your Raspberry Pi logs to it you can easily be alerted to a few critical errors. I use this to get alerted to “out of disk space” and “out of memory” errors.

Out of disk space

Set up a new alert, with the following properties:

1. Query: "No space left on device"
2. Trigger if: number of logs in the last 5 minutes is > 0, it’s high severity
3. Schedule: always
4. Notify: add your email address, Ntfy webhook, etc. and customize notification options as desired

Out of memory

Set up another alert, with the following properties:

1. Query: message:"kernel: Out of memory:"
2. Trigger if: number of logs in the last 5 minutes is > 0, it’s high severity
3. Schedule: always
4. Notify: add your email address, Ntfy webhook, etc. and customize notification options as desired

See Also: Considerations for a long-running Raspberry Pi.