Syslog notes

Notes

Introduction

A critical part of the system managers job is monitoring the system. Most Unix systems use SYSLOG to do this. Syslog allows you to encode messages by level and by facility. Levels can be considered various levels of a problem (e.g. warning, error, emergency) whereas facilities are considered to be service areas (e.g. printing, email, network ). Syslog also allows you to forward log entries to another machine for processing, in this way syslog functions as a distributed error manager.

Levels of Syslog

The levels available are the following:

 LOG_EMERG     A panic condition.  This is normally broadcast to all
                   users.

 LOG_ALERT     A condition that should be corrected immediately, 
                   such as a corrupted system database.

 LOG_CRIT      Critical conditions, e.g., hard device errors.

 LOG_ERR       Errors.

 LOG_WARNING   Warning messages.

 LOG_NOTICE    Conditions that are not error conditions, but should
                   possibly be handled specially.

 LOG_INFO      Informational messages.

 LOG_DEBUG     Messages that contain information normally of 
                   use only when debugging a program.

Facilities that can use syslog

The facilities under which you can log messages are the following:
 LOG_KERN      Messages generated by the kernel.  These cannot be
               generated by any user processes.

 LOG_USER      Messages generated by random user processes.  This is
               the default facility identifier if none is specified.

 LOG_MAIL      The mail system.

 LOG_DAEMON    System daemons, such as routed(1M), ftpd(1M), rshd(1M),
               etc.

 LOG_AUTH      The authorization system:  login(1), su(1M), getty(1M),
               etc.  ftpd(1M), and rshd(1M) also use LOG_AUTH.

 LOG_LPR       The line printer spooling system:  lpr(1), lpd(1M), etc.

 LOG_LOCAL0    Reserved for local use.  Similarly for LOG_LOCAL1 through
               LOG_LOCAL7.

Startup and configuration of syslog

Syslog runs as a deamon named syslogd. This deamon is generally started up in the early stages of multi-user bootup. The syslog reads it configuration information whenever it receives the HUP signal. The file generally is named /etc/syslog.conf.

Syslog uses notation of facility.level, The keyword LOG_ can be ignored. The second part of the line determines what SYSLOG does with the message. That is based on what the starting character is, below is a summary:

     /  A filename (beginning with a leading slash).  The file will be opened
        in append mode.

     @  A hostname preceded by an at sign (``@'').  Selected messages are
        forwarded to the syslogd on the named host.

 Letter A comma-separated list of users.  Selected messages are written to
        those users if they are logged in.

     *  An asterisk.  Selected messages are written to all logged-in users.

     |  A |, followed immediately by a program name, which is taken to be all
        chars after the | up to the next tab; at least one action must follow
        the tab.  The filter is expected to read stdin, and write the filtered
        response to stdout.  If the filter exits with a non-zero value, the
        original message is logged, as well as a message that the filter
        failed.  The filter has a limited time (currently 8 seconds) to
        process the message.  If the filter exits with status 0 without
        writing any data, no message is logged.  The data to be read by the
        filter is not terminated with a newline, nor should the data written
        have a newline appended. 

A sample file might look like this:

# SGI distributed syslog.conf file
#
# Formats: selectoraction
#          selectorfilteraction
kern.debug              |/usr/adm/klogpp        /usr/adm/SYSLOG
#kern.err;user.info;auth.info;lpr.notice;mail.debug     @loghost
*.debug                                                 @loghost
*.debug;user.none;auth.none;local1.none;local2.none;lpr.notice;mail.debug
/usr/adm/SYSLOG
local2.debug            /usr/adm/aguslog
kern.none               /usr/adm/SYSLOG
The level none may be used to disable a facility. This is usually done in the context of eliminating messages, for example the line: *.debug;mail.none /usr/adm/SYSLOG selects debug messages from all facilities except those from mail.

Examples of the types of messages generated

Syslog is a very powerful facility that allows you to sort messages. Below are some of the types of messages you might see in a syslog file.
Feb 27 01:01:04 umbc9 syslogd: restart
Feb 27 01:01:14 umbc9 telnetd[1803]: connect from annex3.umbc.edu
Feb 27 01:02:15 umbc9 rlogind[1845]: connect from annex1.umbc.edu
Feb 27 01:02:44 umbc9 lpd[1879]: /usr/adm/acsps-errs: No such file or 
  directory
Feb 27 01:07:08 umbc9 telnetd[1914]: connect from annex1.umbc.edu
Feb 27 01:08:06 umbc9 rlogind[1946]: connect from annex1.umbc.edu
Feb 27 01:10:28 umbc9 rshd[1985]: connect from xxxx@deputy.cs.umbc.edu
Feb 27 01:10:30 umbc9 rlogind[1993]: connect from xxxx@deputy.cs.umbc.edu
Feb 27 01:13:01 umbc9 sendmail[2042]: BAA02041: to=xzy@picard.cs.wisc.edu,
 delay=00:00:02, mailer=nullclient, relay=mailhub1.gl.umbc.edu. (130.85.3.11),
 stat=Sent (BAA04370 Message accepted for delivery)
Feb 27 02:10:33 umbc9 in.fingerd[3180]: connect from xxx1@umbc9.umbc.edu
Feb 27 02:10:58 umbc9 in.fingerd[3185]: connect from xxx1@umbc9.umbc.edu
Feb 27 02:12:30 umbc9 in.fingerd[3202]: connect from xxx1@umbc9.umbc.edu
Feb 27 02:27:28 umbc9 telnetd[3818]: connect from xxxx@e2-umbc8.umbc.edu
Feb 27 02:39:31 umbc9 in.fingerd[4023]: connect from mcl.mcl.ucsb.edu
Feb 27 05:00:49 umbc9 fsr[4263]:  %frag  after   %free  after  npass  fs
Feb 27 05:00:49 umbc9 fsr[4263]:   1.16   1.02    0.16   0.05      7  
 /dev/root
Feb 27 05:00:49 umbc9 fsr[4263]:    NaN    NaN    0.00   0.00      7  
 /dev/dsk/ dks1d1s6
Feb 27 05:00:49 umbc9 fsr[4263]:   0.72   0.24    0.01   0.01      7  
 /dev/dsk/ dks1d1s1
Feb 27 05:00:49 umbc9 fsr[4263]:   1.78   1.77    0.03   0.02      7  
 /dev/usr
Feb 27 05:00:49 umbc9 fsr[4263]:   1.48   1.48    0.02   0.02      8  
 /dev/dsk/ dks1d2s7
Feb 27 15:18:14 umbc9 ypxfr[10239]: Can't get master of mail.aliases. Reason: 
  no such map in server's domain.
The format of messages is as follows:

Date Time hostname application message

The first message always logged is by the syslog deamon itself places a time stamp on when the deamon was started. The connect messages that are logged after than are generated by a public domain package we have installed named tcpwrapper. This package logs all telnet and rlogin connections. The number in the brackets is the process id of the person on our system. With that we can track what someone has done via system accounting. The mail system, sendmail logs a large number of messages. While this information can be extremely large it comes in very handy when diagnosing mail problems or tracking mail forgeries. Often system utilities log informational information to syslog. The fsr utility de-fragments the file system attempting to make files as contiguous as possible.

Other Important Files Where Information is Logged

Unix has a large number of files that hold logging information. Most of these files are stored in /var/adm. Table 12.1 on Pg. 204 lists a number of these log files and gives a brief description of what they log. I will summarize a few of the important ones.
/var/adm/messages
The messages file holds information that prints to the console. These might include root logins and su attempts. This file has generally been superceded by the SYSLOG facility and may not be found.
/var/adm/lastlog
This file holds the most recent login time for each user in the system. On some systems this is a directory with each user stored as a single record. The directory hold a record for each person that has ever logged in. It can be used to track login records.
The utmp and wtmp files.
The utmp file is where information such as the terminal line, login time, and command executing are stored for access by the who command. The wtmp keeps track of logins and logouts since reboot. The command last reads that file and processes the information.
/var/adm/acct
This is the system accounting file. If enabled, the accounting file records a record for every process listing the following information:
Logging is a critical responsibility of the system administrator. However, in large modern system the amount of information generated can overwhelm most system administrators. Most system administrators develop scritps or tools to examine the log files and extract the important information. This is necessary; however a comprehensive method for monitoring distributed Unix systems is long overdue and a worthy research goal.