Fixing excessive Pi-hole lighttpd access log size when Netdata is installed

November 08, 2023 • Tagged: pihole netdata series:pi-reliability raspberry-pi linux

Part of the Raspberry Pi Reliability series.

Not running out of disk space is an important aspect of keeping tiny Linux machines with limited storage and RAM running reliably for extended time periods.

On most Raspberry Pis and similar devices, I keep /var/log in RAM, either with a straightforward tmpfs mount or with Armbian’s poorly-documented ramlog feature. Keeping these ramdisks from running out of space can be tricky.

I plan to write a future post about the relevant logging and log rotation settings, monitoring for disk space issues, and generally expanding on how my Raspberry Pi log management best practices have changed since this post. But for today, I have one simple tip to address one cause of excessive log growth.

The Problem

I was alerted a couple days ago that /var/log on my home Pi-hole DNS server was out of disk space. On investigation, the main issue was that /var/log/lighttpd/access-pihole.log had grown to more than 20 MB in size. This was odd, since I access the Pi-hole admin UI infrequently and therefore expected this log to be trivially small.

The log was filled with entries like:

1699460409|127.0.0.1|127.0.0.1|GET /admin/api.php?auth=0123456789abcdef0123456789abcdef0123456789abcdef&getQueryTypes=true HTTP/1.1|200|187
1699460409|127.0.0.1|127.0.0.1|GET /admin/api.php?auth=0123456789abcdef0123456789abcdef0123456789abcdef&summaryRaw=true HTTP/1.1|200|709
1699460414|127.0.0.1|127.0.0.1|GET /admin/api.php?auth=0123456789abcdef0123456789abcdef0123456789abcdef&getForwardDestinations=true HTTP/1.1|200|262
1699460414|127.0.0.1|127.0.0.1|GET /admin/api.php?auth=0123456789abcdef0123456789abcdef0123456789abcdef&getQueryTypes=true HTTP/1.1|200|187
1699460414|127.0.0.1|127.0.0.1|GET /admin/api.php?auth=0123456789abcdef0123456789abcdef0123456789abcdef&summaryRaw=true HTTP/1.1|200|709
1699460419|127.0.0.1|127.0.0.1|GET /admin/api.php?auth=0123456789abcdef0123456789abcdef0123456789abcdef&getForwardDestinations=true HTTP/1.1|200|262

Investigation revealed that Netdata was making these queries every 5 seconds as part of its built-in Pi-hole monitoring.

There were some similar entries every minute from another machine that uses Uptime Kuma to alert me if the Pi-hole web UI goes down.

The Solution

I considered turning off the lighttpd access log for the Pi-hole web UI, but:

It is nice to have an access log in general
Pi-hole hardcodes the access log configuration in /etc/lighttpd/conf-available/15-pihole-admin.conf, which gets overwritten during software updates and therefore can’t be modified manually

I also considered advanced logrotate tricks, but Armbian’s ramlog feature complicated this approach.

A previous version of this post recommended using grep + tee periodically to clean unwanted log entries. That approach could lead to log corruption and dropped log entries (though I think the risk in this particular situation is fairly low).

lighttpd developer Glenn Strauss emailed to provide an elegant solution that can be implemented easily using only lighttpd configuration. (Thank you!)

Create a file /etc/lighttpd/conf-available/20-pihole-log.conf with the following content:

$HTTP["remote-ip"] == "127.0.0.1" {
  accesslog.filename = ""
}
$HTTP["remote-ip"] == "[::1]" {
  accesslog.filename = ""
}

sudo ln -s /etc/lighttpd/conf-available/20-pihole-log.conf /etc/lighttpd/conf-enabled/20-pihole-log.conf
sudo systemctl restart lighttpd

This configuration tells lighttpd not to log any access requests originating from 127.0.0.1.