Interesting Shorewall Logs

I've been messing around with centralized syslogs and I found one of my boxes logging MBs of "kernel" information. :O The problem started when I moved all the boxes to syslog-ng. It turned out that the intense logging was a repeated 2 or 3 lines. I searched for the particular log and couldn't find much, so I hope this helps anyone else in a similar position.

The log entries come in the form of BANDWIDTH_IN and BANDWIDTH_OUT, prefixed by kernel: [time.inseconds]. And they are thrown into /var/log/syslog, '/var/log/kern.log', and '/var/log/debug'. It's funny, because most of the message boards where users complain about the log spam suggest using log rotate. The issue here is, if you plan on using the net, your logs will fill up in about 30mins causing syslog-ng to crash. In my case, crashing a centralized log server is not on my to-do list.

So who's the culprit? Webmin and Shorewall of course! I'm not sure how it starts, but I have a good guess that the 'bandwidth' logging feature in Webmin should be blamed. I saw a few references that suggested not using it and correlating their suggestion to high-volume logs. So I logged in to Webmin and removed the logging, then made sure to turn off the log facilities in the Shorewall module in Webmin (just in case). But alas, similar to 'heathaj' on the Ubuntu forums, restarting the box or shorewall will fire up the spam engine, again. :(

So how do we know shorewall is to blame? First, to find out the source of BANDWIDTH_IN and BANDWIDTH_OUT take a look at your iptables with a iptables -L, and look for the INPUT, FORWARD, and OUTPUT chains:

Chain FORWARD (policy DROP)
target prot opt source destination
LOG all -- anywhere anywhere LOG level debug prefix `BANDWIDTH_OUT:'
LOG all -- anywhere anywhere LOG level debug prefix `BANDWIDTH_IN:'
dynamic all -- anywhere anywhere state INVALID,NEW
wlan0_fwd all -- anywhere anywhere
eth0_fwd all -- anywhere anywhere
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
Reject all -- anywhere anywhere
reject all -- anywhere anywhere [goto]

Oh, hello spam engine, how'd you get in there? So now take a look at Shorewall configuration files in /etc/shorewall/, take a close look at 'start'. Found you!

run_iptables -I INPUT -i wlan0 -j LOG --log-prefix BANDWIDTH_IN: --log-level debug
run_iptables -I FORWARD -i wlan0 -j LOG --log-prefix BANDWIDTH_IN: --log-level debug
run_iptables -I FORWARD -o wlan0 -j LOG --log-prefix BANDWIDTH_OUT: --log-level debug
run_iptables -I OUTPUT -o wlan0 -j LOG --log-prefix BANDWIDTH_OUT: --log-level debug

According to the Shorewall documentation page, start is called as an extension, after the firewall starts. Fun. After removing these lines, restarting the firewall, restarting the computer, restarting my KFC in the microwave because I left it there too long while writing this rant, my syslog is logging fine and not spamming my log server. IMO, BANDWIDTH_IN and BANDWIDTH_OUT are very poor choices for logging prefixes, shame on those who chose them! :)

After this little investigation I did a few minutes of searching online. It turns out that there have been some complaints about turning off the bandwidth monitoring feature in Webmin, go figure. There is also a nice thread about the issue here. Remember, those 'logging' lines in iptables are added by Webmin to Shorewall when you enable bandwidth monitoring. At this point, with Webmin version 1.500, the disable button (hidden after selecting an interface to log) does not remove the lines from /etc/shorewall/start. You must do it yourself, hope this helps.