CPL Systems 2020
Unix Server Monitoring

PageR monitors all your UNIX servers from a single server.
All UNIX alerts are funnelled into a single management interface
and you can browse on it with your web browser to see what is
happening.

Several PageR Monitored Objects can be used to monitor UNIX servers.

1. 'UNIX SYSLOG' MO (Monitored Object)

We listen for syslog alerts from anywhere on the network. A single MO
will listen for any alerts, or alternatively specific Unix servers
can be targeted. Each Unix host simply needs its etc/syslog.conf file
 to be edited to point messages to us using an extra line which is

*.warn<tab>@<ip_address_of_PageR_PC>

eg *.warn<tab>@192.69.1.27

The syslogd process will need to be stopped and re-started.
Because SYSLOG is part of UNIX,  you have the benefit of a powerful
free monitoring agent running on each UNIX host which you can now
make full use of.

Shell scripts can now use the UNIX "LOGGER" command to send alerts from
scripts, cron jobs etc. For example,

logger -p local0.alert "problem with overnight backup on Server_A"

The text alert message is sent to your mobile phone, pager or email
via PageR.

2(a) 'HOST PROCESS' and 'HOST VOLUME' MO's

These will monitor PROCESSES and DISC SPACE AVAILABILITY
on many platforms including AIX, HP-UX, SUN, LINUX, VMS,
HP3000 etc.

2(b) HOST LOGIN MOs

These are a more general/generic form of the above but which can
be augmented by your own VB scripting if you wish (and have
the time and knowledge to do it -
see Monitored Objects called
HP9000, RS6000 and Generic Unix).

These log in as a user would and will log out again to test system
availability. While it is logged in it can carry out several checks
on the system. These checks are defined in a VB script which in
turn executes the necessary unix shell commands and analyses the
replies. Unix problems we can monitor include

   CRITICAL PROCESSES STILL RUNNING - eg 'sendmail'
   DISC VOLUME FREE-SPACE LOW
   CRON JOB FINISHED
   WEB SERVER SLOW TO RESPOND
   SECURITY PROBLEM
   FILE SIZE EXCEEDS RECOMMENDED LIMIT
   etc.

Additional checking can be added by the user for specific UNIX
applications, provided the user can write shell script wrapped
up in a VB Script.

3. Scanning UNIX LOG FILES.

We use the FTPGET Monitored Object in PageR to
pull down log files at set intervals for scanning.
We then use our DISC FILE Monitored Object to pick
up new text records detected in the log file. Each
new record detected can be filtered by a string search
to home in on specific problems.

The following command line can be used to write text
records to a log file from any shell script or cron
job:

echo "PageR THERE IS A PROBLEM" >> /etc/alertlog.txt

The above disc file scanning can then be applied to alertlog.txt.

4. TCP Services

Any number of services (or IP 'ports') can be monitored using
the PageR TCP SERVICES Monitored Object.
The user selects from a drop-down list the services to be
monitored. For example
 
     FTP service on port 21
     TELNET service on port 23
     SMTP service on port 25
     HTTP service on port 80
     SUNRPC service on port 111
     SNMP service on port 161
     LPD service on port 515
 
If any of the services fails then an alert is generated.
Any number of services (or 'ports') can be monitored. The
user selects from a drop-down list the services to be
monitored (eg FTP service on port 21 and HTTP service on
port 80). If any of the services fails then an alert is
generated. Users own ports can easily be added to the list.

5. SNMP TRAPS

Any SNMP alerts broadcast from the UNIX server can be picked
up by our software. The SNMP BROWSER in PageR will look
for any SNMP-enabled devices and servers on the network.

6. SNMP QUERY

SNMP Query is different to SNMP Trap.With SNMP Query we go out
and ask the SNMP device if it has any alerts. With Trap we simply
listen for alerts coming to us.

SNMP QUERY depends on the MIBS database we have built into PageR,
which in fact is very comprehensive, and new ones are being added all
the time.

7. IP Ping MO

This is a simple test of availability. The Ping MO can be customised
for number of attempts and timeouts to reduce fale alarms due to
high network traffic.

8. WEB PAGE MONITOR

This is an excellent test of web-page availablility.
User supplies the URL and the timeout value. If we cannot read the
target WEB page within the timeout period then an alert is triggered.
The page content can also be scanned with a string search.

Syslog Monitoring

Setting up syslog for PageR:-

1) On each Unix host to be monitored, use vi to edit
the file /etc/syslog.conf and insert the following line

*.warn <tab> @192.69.1.27

assuming that PageR for NT is on 192.69.1.27

2) Stop the syslog daemon using the 'kill' unix command
on the syslog process id number. It may re-start itself.
Use ps -ef | grep syslog to see the syslogd process.

To start it manually type /etc/syslogd

3) Configure the syslog Monitored Object in PageR
This is very easy, so it is not described here, except
to note the following 'best' settings in the syslog MO:-

(i) Check Log all and all the TOP row of syslog alert boxes
(ii) Use [SENDER] [MSG] as the Alarm text message.

It is only necessary to set up one syslog MO to listen
for syslog alerts from multiple UNIX servers, BUT you
can set up multiple MO's if you wish to have different
alerts for each server.

4) Use the unix 'logger' command to send syslog messages
from shell scripts to PageR like this:-

 $ logger -p local0.alert "UNIX_SERVER_1 has a problem"

Type 'man syslog' and 'man logger'
for more help in unix. See also the file syslog.h
which contains the syntax definitions.

Unix  ‘syslog’ Monitored Object


The ‘syslog’ facility is standard in all Unix servers. The ‘PageR Room Alert’ software has a monitored object which ‘listens’ for syslog alerts. The Unix administrator must setup the syslog subsystem to send syslog warnings and errors to the NT/2K/XP/2003 server or workstation  where the PageR software is installed.

 

Type MAN SYSLOG and MAN LOGGER on Unix for help. See also the file

/usr/include/sys/syslog.h which contains the syntax definitions for logger.

 Syslog Configuration file 


When syslogd daemon is started it looks in

 /etc/syslog.conf  


for its setup information. The following is an example from an HP9000 (HP-UX) system:

 

# @(#) $Revision: 74.1 $

# syslogd configuration file.

# See syslogd(1M) for information about the format of this file.

mail.debug                    /var/adm/syslog/mail.log

*.info;mail.none            /var/adm/syslog/syslog.log

*.info                            @151.198.66.34

 NOTE: there must be a <tab> character between the first and second columns. 


In this example, all levels of event at info or above are directed to the ip address of the server running PageR Room Alert.

 

The following section describes the general features of  the ‘syslog’ subsystem on Unix as it is used by PageR Room Alert. Note that NONE of the syslog features on the UNIX server are part of the PageR product.

 

syslog is a powerful and easily configurable UNIX system resource. Available since the earliest releases of BSD UNIX, it is now supported on most versions of UNIX. Designed to be the UNIX system logging facility, syslog has always offered not just local logging to files, but also remote logging over the network via standard protocols to mixed vendor systems.

Log All Messages Received

Enable to record all Syslog messages received on the Activity Log window even if the messages do not result in an alarm.

Alarm on Message Level

Identify the Syslog message priority levels (0-7) that should generate an alarm.

Apply Search String File to Messages and Alarm on Matches

Enter or select a Search String File name to have each Syslog message searched for any matches to strings or words defined in the search string file. Any match generates an alarm. String search is only performed if the message level does not generate an alarm.

Alarm Object

Identifies the Alarm Object to be used for alarm notification when this monitored object generates an alarm. The drop-down list shows all available Alarm Objects. An Alarm Object must be selected to perform paging, broadcasting or email of alarm events for this object.

Alarm Text

When an alarm is generated for an object, a default alarm notification message is issued by PageR. This message identifies the object and describes the alarm. You can override the default alarm message by entering custom alarm notification message text in this box. You can use substitution keywords in the message which will be replaced by their run time values when the message is generated. Keywords appear as [keyword] in the message text. The keywords you can use for this object are:

 Keyword   Description

[TYPE]      expands to the monitored object's type.

[ID]            expands to the monitored object's unique identification string.

[DESC]                  expands to the monitored object's long description.

[ALARMID]          expands to the unique numeric identifier for the monitored                                              object's current alarm event.

[SENDER]             expands to the IP address of the sending system.

[MSG]                   expands to the text of the Syslog message.

[TIME]                  expands to the current time.

[DATE]                  expands to the current date.

[AGENT]               expands to the the application name of "PageR".

[SYSTEM]            expands to the name of this system.

Notes

Syslog is a logging facility used on all Unix systems including Linux. It is a central message logging tool used by applications and operating systems. One of Syslog's features is the forwarding of messages to other systems over the network.

This is done by adding a line such as

*.info        <tab>         @192.69.1.7          

to the file /etc/syslog.conf on each Unix server to be monitored.

You can configure Syslog to forward messages to the NT/2K/XP/2003 system on which PageR is running and this monitored object will receive and process them.

In this manner, PageR can be used to monitor Unix  systems and other systems that employ Syslog. This monitored object is of the Server/listener type in that is does not perform any "scanning" function. Instead, this object creates a server that listens on the network for Syslog messages and processes them when they arrive. This means that SYSLOG alerts get sent immediately and do not wait for PageR to scan the Unix server which has the problem.

Ther are 3 elements to the syslog system:

1) syslogd daemon (configured with /etc/syslog.conf)

2) syslog program procedure in C

3) syslog ‘logger’ command line interface 

Using syslog  QUICK OVERVIEW


Setting up syslog for PageR:-

 

1) On each Unix host to be monitored, use vi to edit the file /etc/syslog.conf

 

Add a new line like this example,

 

*.info <tab> @192.69.1.27

 

Assuming that PageR Room Alert is on 192.69.1.27

 

2) Stop the syslog daemon syslogd using the 'kill' unix command on the syslog process id number; syslogd should automatically re-start. (use ps -ef | grep syslog to see the process). If it does not re-start automatically type /etc/syslogd at the command line prompt.

 

3) Configure the syslog Monitored Object in PageR

 

Use the unix 'logger' command to send syslog messages, for example

 

logger -p local0.warn “There is a UNIX problem”

 

Creating syslog messages with syslog(3c) and logger(1)

Software developers will create syslog messages through the standard interfaces syslog(3c) and logger(1). The syslog library functions openlog(), closelog(), and setlogmask() are contained in the libc library; function prototypes are defined in syslog.h.

Other users will use the ‘logger’ command (/usr/bin/logger) which provides similar functionality from the command line prompt or in a shell script.

The logger UNIX command


This command can be used online or in shell scripts or batch jobs to report problems to syslog, and via syslog to PageR Room Alert.

Typical command line examples would be

$ logger -p local0.alert “There is a UNIX problem”$ logger -p local0.warn “There is a UNIX problem”


These commands send the message in quotes to the syslog daemon, which then transmits it on according to the setup in /etc/syslog.conf.

logger parses the command line arguments and calls syslog(3c).

 logger Syntax logger [-t tag] [-p priority] [-i] [-f file | message ] 

-t

tag

mark entry with this tag (default is getlogin() )

-p priority

facility.level pair

-i

log the process ID with each message

-f

log from the contents of the file message or log from stdin

For more information on logger type “man logger” at the UNIX prompt.

 The syslog C call for UNIX C programmers

The syslog call provides an interface with printf()-like string handling to syslog.

Here is an example:

syslog(priority,"message_string")

Message priority is a combination of a syslog level and facility tag. The message string has a printf-like interface: one can log ASCII strings using standard % format semantics. In addition, the "%m" format will print the global errno for the originating process. syslog() will return an error if /dev/log cannot be opened. For the 10.x releases syslog() was modified to retry the write to /dev/log if it is busy, up to 20 times with a 1/10th -second delay.

The function openlog() provides a useful method for controlling the logging process and setting default fields in the message strings. The function is called as follows:

openlog("identifying_string",log_options,default_facility)

where the following parameters can be set:

"identifying_string" is a string that is prepended to each syslog message

log_options can be one or more of the following:

LOG_PID

log process ID with each message (very useful)

LOG_CONS

write to console if unable to send to syslogd

LOG_NDELAY

open connection to syslogd immediately

LOG_NOWAIT

no wait for children logging messages to console

default_facility assigns a facility tag to identify the originating subsystem (very useful)

The following calls are also part of the syslog library interface:

setlogmask(maskpri)--sets the process log priority mask to maskpri and returns the previous priority mask value

closelog()--closes the log file

Example code

A developer first configures default settings for message strings using openlog(), and then logs messages to syslog() as needed.

Configure the log file:

openlog("Reactor control subsystem",LOG_PID|LOG_CONS,LOG_LOCAL0);

Send a message:

syslog(LOG_ALERT,"meltdown imminent!"); 

Configuring syslog with /etc/syslog.conf


When the operating system is booted, syslogd is brought up as a user space process. The process syslogd is started early in the start sequence so that subsequent services can use syslog logging if needed.

When the syslogd daemon is started, it reads the configuration file

/etc/syslog.conf

for its initial configuration. One can reconfigure syslogd and cause it to re-read its configuration file by sending it the SIGHUP signal (kill -1 syslogd_pid). The rules in /etc/syslog.conf are set by the system administrator.

Each non-comment line of /etc/syslog.conf is read as a configuration directive.

Priority tags can be one of the following ASCII strings; these strings correspond to the priority tags defined earlier for the syslog(3c) interface:

emerg or panic, alert, crit, error or err, warning or warn notice, info, debug

Similarly, the facility tag can be one of the following strings, also corresponding to the facility tags defined earlier for syslog(3c):

kern, user, mail, daemon, auth or security, mark, syslog, lp or lpr, local[0-7]

Examples of possible selectors include:

*.emerg <tab> @ntserver.workgroup.com # emergencies to PageR

The action field specifies where the message is to be directed, for example:

@remote_hostforward to syslogd on a remote system eg running PageR Room Alert

file name

can be a regular text file, for example a log file on the Unix server

device file

for example, the console at /dev/console

a list of users

write to a user if logged in

*

write to all users currently logged in

Here are further examples examples of selectors one might use:

*.alert <tab> @192.69.1.27 # re-direct messages to PageR

*.emerg <tab> /dev/console # might be crashing!

*.emerg <tab> * # alert all users currently logged in

*.alert <tab> root,fenwick # let root know ALERTS

*.info <tab> /var/adm/syslog/syslog.log # standard log file

The <tab> character (ASCII tab) is important; in HP-UX this separator must be the tab character, and not a blank character or any other whitespace. Starting with the HP-UX 10.x releases, the standard location for logging syslog messages is the file /var/adm/syslog/syslog.log. The log file from the immediate prior bootup or invocation of syslog is preserved in the file /var/adm/syslog/OLDsyslog.log.

More on Logging to Remote Hosts

One of the most useful features of syslog is that syslogd can write to an Internet-domain socket connected to a socket on a remote host. The local syslogd daemon opens a socket and writes messages to it. A remote syslogd daemon receives the message. This socket is bound to Port 514/udp, which is reserved for syslog in /etc/services. Multiple hosts can be specified, and remote hosts can be chained together. If one is chaining remote hosts together, one should be aware that the logged message will list the system name of the sending system (which may or may not be the originating).

Here is an example of how one would configure syslog for remote logging to the PageR NT/2K/XP/2003 system "admin" in the file /etc/syslog.conf:

*.error <tab>  @admin.hp.com # forward to PageR NT/2K/XP/2003 workstation 

System-generated syslog messages

Many UNIX subsystems have been converted to log their messages through syslog. One can now review most kernel-generated messages through the syslog logfile. Kernel-generated messages are routed through the character-special device file /dev/klog. Most messages generated by the kernel in bootup and operation are logged to syslog. In fact, most of the information presented on the kernel bootup screens is logged. One can find the same information in the syslog log file that one can find in the kernel message buffer--there is no longer any reason to use the dmesg(1m) command.

Many of the Internet services (such as the inetd server and sendmail) have also been configured to log errors to syslog. The nettl logging service used in network drivers and services is a different logging routine, but it announces its own startup through syslog.

Since syslog operates over TCP/IP networks, networked peripherals can also use syslog services. For example, HP printers that use the JetDirect network interface card can be configured to report error conditions to syslog. This may be a convenient way to detect and report common printer errors.