Assume you have a number of patterns you might want to watch out for in various logfiles. These could be harmless, e.g. postfix/smtp\[[[:digit:]]+\]: .* status=sent which indicates that a mail was sent by postfix, or could be bad, e.g. indications of hardware troubles, firewall log entries or suspicious entries in proxy logs.
There are a number of tools which can help you do that, e.g. logwatch can produce nice reports for you.
In my case, I was less interested in individual log entries, but in a graphical representation of their frequency. For that, I need something similar to MRTG. Cacti is a decent graphical frontend to the core MRTG engine, rrdtool. It provides various data sources and graphing options all packed up into a neat web interface.
So, what needs to be done to get Cacti to graph match-rates of regexs in logfiles?
First of all, we need code which reads logfiles, copes with the various log-rotation schemes and does the actual pattern matching. Luckily, a rarely used feature of the net-snmp snmpd does exactly that. From the man-page of snmpd.conf
logmatch NAME PATH CYCLETIME REGEX
monitors the specified file for occurances of the specified pattern REGEX.
A maximum of 50 files can be monitored.
Perfect! Especially as querying SNMP data is a bread-and-butter job for Cacti.
The thing you have to watch out for are file permissions: snmpd needs to be able to actually read the files you told him to monitor. On my Debian box, the defaults prevent this. My solution is to tell snmpd to change its group to adm when starting: Edit /etc/default/snmpd and add a “-g 4” to SNMPDOPTS.
There is a good step-by-step guide on the Cacti homepage on how to get Cacti to graph SNMP tables. First of all, you need an XML file which describes the MIB you want Cacti to query:
Put Cacti Data Source definition for logmatch in /usr/share/cacti/resource/snmp_queries/ (this is for Debian).
Then you need to add various templates to the system to get the data collection and graphing up and running. Either follow the guide or import cacti_data_query_ucdnet_-_logmatch.xml (this is for Cacti 7.8b). Enable this data-source in the “device” page and create graphs, and soon you get something like:
Here are all the files (plus a snmpd.conf snipped) in a .tar file: Cacti files for logmatch
Share and Enjoy!