I like to go through my system logs. This often lets me know if
there's a system problem before it becomes a major issue, and it shows
me security issues.
Reviewing system logs has problems:
log_analysis is my solution to these problems. It goes through several
different kinds of logs (currently syslog, wtmp, and sulog), over some
period (defaults to yesterday). It strips out the date and PID, and
throws away certain entries. Then it tries each entry against a list
of perl regular expressions. Each perl regular expression is
associated with a category name and a rule for extracting data. When
there's a match, the data-extracting rule is applied, and filed under
the category. If a log entry is unknown, it's filed under a special
category for unknowns. Identical entries for a given category are
sorted and counted. There's an option to mail the output, so you can
just run it out of cron. You can also save a local copy of the
output. If you prefer to PGP-mail yourself the output, you can do
this, too. The whole thing is designed to be easily extended, complete
with an easy plug-in interface. The default mode is for reporting, but
it also "real" and "gui" modes for continuous monitoring, complete with
action support. Oh, and you can edit patterns in a GUI that helps write
regular expressions quickly and easily.
- Logs contain lots of extraneous stuff that I want to be logged,
but that I don't want to sift through when I review logs (ie. routine,
error-free daemon operation.)
- Logs contain a lot of repetition, which drowns out the
- Noting repetition can be tricky because each entry usually has
extra features to make it unique, such as a date, maybe a PID (ie. for
syslog), and maybe application-specific information (ie. sendmail
- One needs to remember to review them. :)
- One needs to be root to looks at logs for some OSs.
- On most systems, looking at the logs for just one day can be a
- If I attack each box I deal with and write a separate script to do
all this, I'll waste a lot of time duplicating effort.
- Writing patterns is a pain even if you know regular expressions.
The program needs to run with permissions to
read your log files in order to be useful, which usually means root.
It does not default to SUID root, and I recommend not making it SUID,
so just run it as root (ie. manually or out of cron). I've tried to
avoid temp files everywhere that I can, and in the one case where I do
use a temp file, I make sure to use the POSIX tmpnam function instead
of trying to make up my own temp file algorithm. The default umask is
077. If you use action commands, there is nothing to stop you from
using parts of the log message in insecure ways, so for goodness' sake,
log_analysis already has lots of rules,
but chances are that you have log entries that aren't already covered.
So, log_analysis can easily be extended via a local config file, as
documented in the log_analysis manpage. There's even an easy way to do
We have a mailing list. It's run by majordomo, off the frakir.org
server, and it's called log_analysis. That should give you enough info
to figure out how to subscribe. :)
Here is the ChangeLog.
- no more gui_mode_configuration_disabled or gui_mode_ignore_disabled
- using the "gui_mode_config_savelocal" option will not recognize
local modifications made from earlier versions of log_analysis.
- dests may no longer contain backslash
- date_format defaults to %Y_%m_%d
- -o no longer also outputs to standard out. Add -O for the old
- config_version is now mandatory
- end has been replaced with @@end
- the new default sort is "funky". It handles numbers and IPs
more cleanly, although it takes longer for large data sets.
- -F has been changed to -I internal_config
- -D has been changed to -I evals
- raw_rules no longer allows an optional code hook as its
4th field. If you were using this, please revert to the old
version and let me know.
- Took required_log_file out of the config; supporting it
was too messy, and everyone I asked wasn't using it, anyway.
If you were using it, please revert to the old version and let
- Some directive names used "-" instead of "_". These have been
changed. The new names are include_dir,
Major things I would really like to see happen, but don't necessarily have
time to implement in the foreseeable future:
- Statistics support. That is, some way to keep track of information beyond
simple message counts -- say, so we can know not only how many mail
message lines we've seen, but also how many bytes those messages
contained. This probably includes the ability to allow each
message to count towards multiple categories.
- Persistence. This goes with statistics -- once I can know, say, many
bytes I get per day, I would now like a way for log_analysis to keep track
of that, and tell me how many we saw this month or this year.
- Support for "state" across lines. That is, some kinds of log messages
(most notably sendmail) don't provide all the relevant info in each log
message, so you need to keep track of previous log messages. There is
no mechanism to handle this.
- A config syntax sophisticated enough to keep up with all this.
- This page sucks. :) Rewrite to be more useful.
by Mordechai T. Abzug