Unix  ‘syslog’ Monitored Object

The ‘syslog’ facility is standard in all Unix servers. The ‘PageR Room Alert’ software has a monitored object which ‘listens’ for syslog alerts. The Unix administrator must setup the syslog subsystem to send syslog warnings and errors to the NT/2K/XP/2003 server or workstation  where the PageR software is installed.

 

Type MAN SYSLOG and MAN LOGGER on Unix for help. See also the file

/usr/include/sys/syslog.h which contains the syntax definitions for logger.

 Syslog Configuration file 

When syslogd daemon is started it looks in

 /etc/syslog.conf  

for its setup information. The following is an example from an HP9000 (HP-UX) system:

 

# @(#) $Revision: 74.1 $

# syslogd configuration file.

# See syslogd(1M) for information about the format of this file.

mail.debug                    /var/adm/syslog/mail.log

*.info;mail.none            /var/adm/syslog/syslog.log

*.info                            @151.198.66.34

 NOTE: there must be a <tab> character between the first and second columns. 

In this example, all levels of event at info or above are directed to the ip address of the server running PageR Room Alert.

 

The following section describes the general features of  the ‘syslog’ subsystem on Unix as it is used by PageR Room Alert. Note that NONE of the syslog features on the UNIX server are part of the PageR product.

 

syslog is a powerful and easily configurable UNIX system resource. Available since the earliest releases of BSD UNIX, it is now supported on most versions of UNIX. Designed to be the UNIX system logging facility, syslog has always offered not just local logging to files, but also remote logging over the network via standard protocols to mixed vendor systems.

Log All Messages Received

Enable to record all Syslog messages received on the Activity Log window even if the messages do not result in an alarm.

Alarm on Message Level

Identify the Syslog message priority levels (0-7) that should generate an alarm.

Apply Search String File to Messages and Alarm on Matches

Enter or select a Search String File name to have each Syslog message searched for any matches to strings or words defined in the search string file. Any match generates an alarm. String search is only performed if the message level does not generate an alarm.

Alarm Object

Identifies the Alarm Object to be used for alarm notification when this monitored object generates an alarm. The drop-down list shows all available Alarm Objects. An Alarm Object must be selected to perform paging, broadcasting or email of alarm events for this object.

Alarm Text

When an alarm is generated for an object, a default alarm notification message is issued by PageR. This message identifies the object and describes the alarm. You can override the default alarm message by entering custom alarm notification message text in this box. You can use substitution keywords in the message which will be replaced by their run time values when the message is generated. Keywords appear as [keyword] in the message text. The keywords you can use for this object are:

 Keyword   Description

[TYPE]      expands to the monitored object's type.

[ID]            expands to the monitored object's unique identification string.

[DESC]                  expands to the monitored object's long description.

[ALARMID]          expands to the unique numeric identifier for the monitored                                              object's current alarm event.

[SENDER]             expands to the IP address of the sending system.

[MSG]                   expands to the text of the Syslog message.

[TIME]                  expands to the current time.

[DATE]                  expands to the current date.

[AGENT]               expands to the the application name of "PageR".

[SYSTEM]            expands to the name of this system.

Notes

Syslog is a logging facility used on all Unix systems including Linux. It is a central message logging tool used by applications and operating systems. One of Syslog's features is the forwarding of messages to other systems over the network.

This is done by adding a line such as

*.info        <tab>         @192.69.1.7          

to the file /etc/syslog.conf on each Unix server to be monitored.

You can configure Syslog to forward messages to the NT/2K/XP/2003 system on which PageR is running and this monitored object will receive and process them.

In this manner, PageR can be used to monitor Unix  systems and other systems that employ Syslog. This monitored object is of the Server/listener type in that is does not perform any "scanning" function. Instead, this object creates a server that listens on the network for Syslog messages and processes them when they arrive. This means that SYSLOG alerts get sent immediately and do not wait for PageR to scan the Unix server which has the problem.

Ther are 3 elements to the syslog system:

1) syslogd daemon (configured with /etc/syslog.conf)

2) syslog program procedure in C

3) syslog ‘logger’ command line interface 

Using syslog  QUICK OVERVIEW

Setting up syslog for PageR:-

 

1) On each Unix host to be monitored, use vi to edit the file /etc/syslog.conf

 

Add a new line like this example,

 

*.info <tab> @192.69.1.27

 

Assuming that PageR Room Alert is on 192.69.1.27

 

2) Stop the syslog daemon syslogd using the 'kill' unix command on the syslog process id number; syslogd should automatically re-start. (use ps -ef | grep syslog to see the process). If it does not re-start automatically type /etc/syslogd at the command line prompt.

 

3) Configure the syslog Monitored Object in PageR

 

Use the unix 'logger' command to send syslog messages, for example

 

logger -p local0.warn “There is a UNIX problem”

 

Creating syslog messages with syslog(3c) and logger(1)

Software developers will create syslog messages through the standard interfaces syslog(3c) and logger(1). The syslog library functions openlog(), closelog(), and setlogmask() are contained in the libc library; function prototypes are defined in syslog.h.

Other users will use the ‘logger’ command (/usr/bin/logger) which provides similar functionality from the command line prompt or in a shell script.

The logger UNIX command

This command can be used online or in shell scripts or batch jobs to report problems to syslog, and via syslog to PageR Room Alert.

Typical command line examples would be

$ logger -p local0.alert “There is a UNIX problem”$ logger -p local0.warn “There is a UNIX problem”

These commands send the message in quotes to the syslog daemon, which then transmits it on according to the setup in /etc/syslog.conf.

logger parses the command line arguments and calls syslog(3c).

 logger Syntax logger [-t tag] [-p priority] [-i] [-f file | message ] 

-t tag	mark entry with this tag (default is getlogin() )
-p priority	facility.level pair
-i	log the process ID with each message
-f	log from the contents of the file message or log from stdin

For more information on logger type “man logger” at the UNIX prompt.

 The syslog C call for UNIX C programmers

The syslog call provides an interface with printf()-like string handling to syslog.

Here is an example:

syslog(priority,"message_string")

Message priority is a combination of a syslog level and facility tag. The message string has a printf-like interface: one can log ASCII strings using standard % format semantics. In addition, the "%m" format will print the global errno for the originating process. syslog() will return an error if /dev/log cannot be opened. For the 10.x releases syslog() was modified to retry the write to /dev/log if it is busy, up to 20 times with a ¹/₁₀th -second delay.

The function openlog() provides a useful method for controlling the logging process and setting default fields in the message strings. The function is called as follows:

openlog("identifying_string",log_options,default_facility)

where the following parameters can be set:

"identifying_string" is a string that is prepended to each syslog message

log_options can be one or more of the following:

LOG_PID	log process ID with each message (very useful)
LOG_CONS	write to console if unable to send to syslogd
LOG_NDELAY	open connection to syslogd immediately
LOG_NOWAIT	no wait for children logging messages to console

default_facility assigns a facility tag to identify the originating subsystem (very useful)

The following calls are also part of the syslog library interface:

setlogmask(maskpri)--sets the process log priority mask to maskpri and returns the previous priority mask value

closelog()--closes the log file

Example code

A developer first configures default settings for message strings using openlog(), and then logs messages to syslog() as needed.

Configure the log file:

openlog("Reactor control subsystem",LOG_PID|LOG_CONS,LOG_LOCAL0);

Send a message:

syslog(LOG_ALERT,"meltdown imminent!"); 

Configuring syslog with /etc/syslog.conf

When the operating system is booted, syslogd is brought up as a user space process. The process syslogd is started early in the start sequence so that subsequent services can use syslog logging if needed.

When the syslogd daemon is started, it reads the configuration file

/etc/syslog.conf

for its initial configuration. One can reconfigure syslogd and cause it to re-read its configuration file by sending it the SIGHUP signal (kill -1 syslogd_pid). The rules in /etc/syslog.conf are set by the system administrator.

Each non-comment line of /etc/syslog.conf is read as a configuration directive.

Priority tags can be one of the following ASCII strings; these strings correspond to the priority tags defined earlier for the syslog(3c) interface:

emerg or panic, alert, crit, error or err, warning or warn notice, info, debug

Similarly, the facility tag can be one of the following strings, also corresponding to the facility tags defined earlier for syslog(3c):

kern, user, mail, daemon, auth or security, mark, syslog, lp or lpr, local[0-7]

Examples of possible selectors include:

*.emerg <tab> @ntserver.workgroup.com # emergencies to PageR

The action field specifies where the message is to be directed, for example:

@remote_host	forward to syslogd on a remote system eg running PageR Room Alert
file name	can be a regular text file, for example a log file on the Unix server
device file	for example, the console at /dev/console
a list of users	write to a user if logged in
*	write to all users currently logged in

Here are further examples examples of selectors one might use:

*.alert <tab> @192.69.1.27 # re-direct messages to PageR

*.emerg <tab> /dev/console # might be crashing!

*.emerg <tab> * # alert all users currently logged in

*.alert <tab> root,fenwick # let root know ALERTS

*.info <tab> /var/adm/syslog/syslog.log # standard log file

The <tab> character (ASCII tab) is important; in HP-UX this separator must be the tab character, and not a blank character or any other whitespace. Starting with the HP-UX 10.x releases, the standard location for logging syslog messages is the file /var/adm/syslog/syslog.log. The log file from the immediate prior bootup or invocation of syslog is preserved in the file /var/adm/syslog/OLDsyslog.log.

More on Logging to Remote Hosts

One of the most useful features of syslog is that syslogd can write to an Internet-domain socket connected to a socket on a remote host. The local syslogd daemon opens a socket and writes messages to it. A remote syslogd daemon receives the message. This socket is bound to Port 514/udp, which is reserved for syslog in /etc/services. Multiple hosts can be specified, and remote hosts can be chained together. If one is chaining remote hosts together, one should be aware that the logged message will list the system name of the sending system (which may or may not be the originating).

Here is an example of how one would configure syslog for remote logging to the PageR NT/2K/XP/2003 system "admin" in the file /etc/syslog.conf:

*.error <tab>  @admin.hp.com # forward to PageR NT/2K/XP/2003 workstation 

System-generated syslog messages

Many UNIX subsystems have been converted to log their messages through syslog. One can now review most kernel-generated messages through the syslog logfile. Kernel-generated messages are routed through the character-special device file /dev/klog. Most messages generated by the kernel in bootup and operation are logged to syslog. In fact, most of the information presented on the kernel bootup screens is logged. One can find the same information in the syslog log file that one can find in the kernel message buffer--there is no longer any reason to use the dmesg(1m) command.

Many of the Internet services (such as the inetd server and sendmail) have also been configured to log errors to syslog. The nettl logging service used in network drivers and services is a different logging routine, but it announces its own startup through syslog.

Since syslog operates over TCP/IP networks, networked peripherals can also use syslog services. For example, HP printers that use the JetDirect network interface card can be configured to report error conditions to syslog. This may be a convenient way to detect and report common printer errors.