PageR monitors all your
UNIX servers from a single server.
All UNIX alerts are funnelled into a single management interface
can browse on it with your web browser to see what is
Several PageR Monitored Objects can be used to
monitor UNIX servers.
1. 'UNIX SYSLOG' MO (Monitored Object)
We listen for syslog alerts from anywhere on the network. A single MO
listen for any alerts, or alternatively specific Unix servers
can be targeted. Each Unix host simply needs its etc/syslog.conf
to be edited to point messages to us using an extra line which is
The syslogd process will need to be stopped and re-started.
Because SYSLOG is part of UNIX, you have the benefit of a powerful
monitoring agent running on each UNIX host which you can now
make full use of.
Shell scripts can now use the UNIX "LOGGER"
command to send alerts from
scripts, cron jobs etc. For example,
logger -p local0.alert "problem with overnight backup on Server_A"
The text alert message is sent to your mobile phone, pager or email
2(a) 'HOST PROCESS' and 'HOST VOLUME' MO's
These will monitor
PROCESSES and DISC SPACE AVAILABILITY
on many platforms including AIX, HP-UX, SUN, LINUX, VMS,
2(b) HOST LOGIN MOs
These are a more general/generic form of the above
but which can
be augmented by your own VB scripting if you wish (and have
the time and knowledge to do it - see Monitored Objects called
HP9000, RS6000 and Generic Unix).
These log in as a user would and will log out again to test system
While it is logged in it can carry out several checks
on the system. These checks are defined in a VB script which in
turn executes the necessary unix shell commands and analyses the
replies. Unix problems we can monitor include
CRITICAL PROCESSES STILL RUNNING - eg 'sendmail'
DISC VOLUME FREE-SPACE LOW
CRON JOB FINISHED
WEB SERVER SLOW TO RESPOND
FILE SIZE EXCEEDS RECOMMENDED LIMIT
Additional checking can be added by the user for
applications, provided the user can write shell script wrapped
up in a VB Script.
3. Scanning UNIX LOG FILES.
We use the FTPGET Monitored Object in PageR to
pull down log files at set intervals for scanning.
then use our DISC FILE Monitored Object to pick
up new text records detected in the log file. Each
new record detected
can be filtered by a string search
to home in on specific problems.
The following command line can be used to write text
records to a log file from any shell script or cron
echo "PageR THERE IS A PROBLEM" >> /etc/alertlog.txt
The above disc
file scanning can then be applied to alertlog.txt.
Any number of services (or IP
'ports') can be monitored using
the PageR TCP SERVICES Monitored Object.
The user selects from a drop-down list the services to be
monitored. For example
FTP service on port 21
TELNET service on port 23
on port 25
HTTP service on port 80
SUNRPC service on port 111
SNMP service on port 161
LPD service on port 515
any of the services fails then an alert is generated.
Any number of services (or 'ports') can be monitored. The
user selects from a drop-down list the services to be
monitored (eg FTP service on port 21 and HTTP service on
80). If any of the services fails then an alert is
generated. Users own ports can easily be added to the list.
5. SNMP TRAPS
Any SNMP alerts broadcast from the UNIX server
can be picked
up by our software. The SNMP BROWSER in PageR will look
for any SNMP-enabled devices and servers on
6. SNMP QUERY
SNMP Query is different to SNMP Trap.With SNMP Query we go out
and ask the
SNMP device if it has any alerts. With Trap we simply
listen for alerts coming to us.
SNMP QUERY depends on the MIBS database we have built into PageR,
fact is very comprehensive, and new ones are being added all
7. IP Ping MO
This is a simple test of availability. The Ping MO can be customised
of attempts and timeouts to reduce fale alarms due to
high network traffic.
8. WEB PAGE MONITOR
This is an excellent test of web-page availablility.
User supplies the URL and the timeout value. If we cannot read the
target WEB page within the timeout period
then an alert is triggered.
The page content can also be scanned with a string search.
Setting up syslog for PageR:-
1) On each Unix host to be monitored, use vi to
the file /etc/syslog.conf and insert the following line
*.warn <tab> @18.104.22.168
assuming that PageR for NT is on 22.214.171.124
2) Stop the
syslog daemon using the 'kill' unix command
on the syslog process id number. It may re-start itself.
ps -ef | grep syslog to see the syslogd process.
start it manually type /etc/syslogd
3) Configure the syslog Monitored Object in PageR
This is very easy, so it
is not described here, except
to note the following 'best' settings in the syslog MO:-
(i) Check Log all and all the TOP row of syslog alert boxes
(ii) Use [SENDER]
[MSG] as the Alarm text message.
It is only necessary
to set up one syslog MO to listen
for syslog alerts from multiple UNIX servers, BUT you
can set up multiple MO's
if you wish to have different
alerts for each server.
4) Use the unix 'logger' command to send syslog messages
from shell scripts to PageR like this:-
$ logger -p local0.alert "UNIX_SERVER_1 has a problem"
Type 'man syslog' and 'man logger'
for more help in
unix. See also the file syslog.h
which contains the syntax definitions.
‘syslog’ Monitored Object
The ‘syslog’ facility is standard in all Unix servers. The ‘PageR Room Alert’
software has a monitored object which ‘listens’ for syslog alerts. The Unix administrator must setup the syslog
subsystem to send syslog warnings and errors to the NT/2K/XP/2003 server or workstation where the PageR
software is installed.
Type MAN SYSLOG and MAN LOGGER on Unix for help. See also the file
/usr/include/sys/syslog.h which contains
the syntax definitions for logger. Syslog Configuration file
When syslogd daemon is started
it looks in
for its setup information. The following is an example
from an HP9000 (HP-UX) system:
# @(#) $Revision: 74.1 $
# syslogd configuration file.
# See syslogd(1M) for information about the format of this
@126.96.36.199 NOTE: there must be a <tab> character between the first and second columns.
In this example, all levels of event at info or above are directed to the ip address of the server running
PageR Room Alert.
The following section describes the general features of the
‘syslog’ subsystem on Unix as it is used by PageR Room Alert. Note that NONE of the syslog features on the UNIX
server are part of the PageR product.
syslog is a powerful
and easily configurable UNIX system resource. Available since the earliest releases of BSD UNIX, it is now supported on most
versions of UNIX. Designed to be the UNIX system logging facility, syslog has always offered not just local logging
to files, but also remote logging over the network via standard protocols to mixed vendor systems.
Log All Messages Received
Enable to record all Syslog messages
received on the Activity Log window even if the messages do not result in an alarm.
Alarm on Message Level
Identify the Syslog message priority levels (0-7) that
should generate an alarm.
Apply Search String File to Messages and Alarm on Matches
Enter or select a Search String File name to have each Syslog message searched
for any matches to strings or words defined in the search string file. Any match generates an alarm. String search is only
performed if the message level does not generate an alarm.
Identifies the Alarm Object to be used for alarm notification when this monitored object generates an
alarm. The drop-down list shows all available Alarm Objects. An Alarm Object must be selected to perform paging, broadcasting
or email of alarm events for this object.
When an alarm is generated for an object, a default alarm notification message is issued by PageR. This message
identifies the object and describes the alarm. You can override the default alarm message by entering custom alarm notification
message text in this box. You can use substitution keywords in the message which will be replaced by their run time values
when the message is generated. Keywords appear as [keyword] in the message text. The keywords you can use for this object
[TYPE] expands to the monitored object's type.
expands to the monitored object's unique identification string.
expands to the monitored object's long description.
[ALARMID] expands to the unique
numeric identifier for the monitored
object's current alarm event.
[SENDER] expands to the
IP address of the sending system.
expands to the text of the Syslog message.
expands to the current time.
expands to the current date.
to the the application name of "PageR".
to the name of this system.
is a logging facility used on all Unix systems including Linux. It is a central message logging tool used by applications
and operating systems. One of Syslog's features is the forwarding of messages to other systems over the network.
This is done by adding a line such
to the file /etc/syslog.conf
on each Unix server to be monitored.
You can configure Syslog to forward messages to the NT/2K/XP/2003 system on which PageR is running and
this monitored object will receive and process them.
In this manner, PageR can be used to monitor Unix systems and other systems
that employ Syslog. This monitored object is of the Server/listener type in that is does not perform any "scanning"
function. Instead, this object creates a server that listens on the network for Syslog messages and processes them when they
arrive. This means that SYSLOG alerts get sent immediately and do not wait for PageR to scan the Unix server which has the
are 3 elements to the syslog system:
1) syslogd daemon (configured with /etc/syslog.conf)
2) syslog program procedure in C
3) syslog ‘logger’
command line interface Using syslog QUICK OVERVIEW
Setting up syslog for PageR:-
1) On each Unix host to be monitored, use vi to edit the file /etc/syslog.conf
Add a new line like this example,
*.info <tab> @188.8.131.52
Assuming that PageR Room Alert is on 184.108.40.206
2) Stop the syslog
daemon syslogd using the 'kill' unix command on the syslog process id number; syslogd should automatically
re-start. (use ps -ef | grep syslog to see the process). If it does not re-start automatically type /etc/syslogd at the command
3) Configure the syslog Monitored Object in PageR
Use the unix 'logger' command to send syslog messages, for example
logger -p local0.warn
“There is a UNIX problem”
Creating syslog messages with syslog(3c)
Software developers will create syslog messages through the standard interfaces
syslog(3c) and logger(1). The syslog library functions openlog(), closelog(), and setlogmask()
are contained in the libc library; function prototypes are defined in syslog.h.
will use the ‘logger’ command (/usr/bin/logger) which provides similar functionality
from the command line prompt or in a shell script.The logger UNIX command
This command can be used online or in shell scripts or
batch jobs to report problems to syslog, and via syslog to PageR Room Alert.
Typical command line examples would be$ logger -p local0.alert “There is a UNIX problem”$ logger -p local0.warn “There is a UNIX problem”
These commands send
the message in quotes to the syslog daemon, which then transmits it on according to the setup in /etc/syslog.conf.
parses the command line arguments and calls syslog(3c). logger Syntax logger [-t tag] [-p priority] [-i] [-f file | message ]
mark entry with this tag (default is getlogin() )
log the process ID with each message
log from the contents of the file message or log from stdin
For more information on logger
type “man logger” at the UNIX prompt. The syslog C call for UNIX C programmers
The syslog call
provides an interface with printf()-like string handling to syslog.
Here is an example:
Message priority is a combination
of a syslog level and facility tag. The message string has a printf-like interface: one can log ASCII strings
using standard % format semantics. In addition, the "%m" format will print the global errno
for the originating process. syslog() will return an error if /dev/log cannot be opened. For the 10.x releases
syslog() was modified to retry the write to /dev/log if it is busy, up to 20 times with a 1/10th
The function openlog() provides a useful method for controlling the logging process and setting default
fields in the message strings. The function is called as follows:
where the following parameters
can be set:
is a string that is prepended to each syslog message
log_options can be one or more of the following:
log process ID
with each message (very useful)
write to console if unable to send
open connection to syslogd immediately
no wait for children logging messages to console
default_facility assigns a facility tag to identify the originating subsystem (very useful)
The following calls are also
part of the syslog library interface:
setlogmask(maskpri)--sets the process log priority mask
to maskpri and returns the previous priority mask value
closelog()--closes the log file
A developer first configures default settings for message strings using openlog(), and then logs messages
to syslog() as needed.
Configure the log file:
openlog("Reactor control subsystem",LOG_PID|LOG_CONS,LOG_LOCAL0);
Send a message:syslog(LOG_ALERT,"meltdown imminent!");
Configuring syslog with /etc/syslog.conf
When the operating system is booted, syslogd
is brought up as a user space process. The process syslogd is started early in the start sequence so that subsequent
services can use syslog logging if needed.
When the syslogd daemon is started, it reads the configuration file /etc/syslog.conf
for its initial configuration.
One can reconfigure syslogd and cause it to re-read its configuration file by sending it the SIGHUP signal (kill -1 syslogd_pid). The rules in /etc/syslog.conf are set by the system administrator.
Each non-comment line of /etc/syslog.conf
is read as a configuration directive.
Priority tags can be one of the following ASCII strings; these
strings correspond to the priority tags defined earlier for the syslog(3c) interface:emerg or panic, alert, crit, error
or err, warning or warn notice, info, debug
Similarly, the facility tag can be one of the following strings, also corresponding
to the facility tags defined earlier for syslog(3c):kern, user, mail, daemon, auth or security, mark, syslog, lp or lpr, local[0-7]
Examples of possible selectors
*.emerg <tab> @ntserver.workgroup.com # emergencies to PageR
The action field specifies where the
message is to be directed, for example:
|@remote_host||forward to syslogd on a remote system eg running PageR Room Alert|
can be a regular text file, for example a log file on the Unix server
for example, the console at /dev/console
a list of users
write to a user if logged in
write to all users currently logged in
Here are further examples
examples of selectors one might use:*.alert <tab> @220.127.116.11 # re-direct messages to PageR
*.emerg <tab> /dev/console # might be crashing!
*.emerg <tab> * # alert all users currently logged in
*.alert <tab> root,fenwick # let root know ALERTS
*.info <tab> /var/adm/syslog/syslog.log # standard log file
The <tab> character (ASCII tab) is important;
in HP-UX this separator must be the tab character, and not a blank character or any other whitespace. Starting with the HP-UX
10.x releases, the standard location for logging syslog messages is the file /var/adm/syslog/syslog.log.
The log file from the immediate prior bootup or invocation of syslog is preserved in the file /var/adm/syslog/OLDsyslog.log.More on Logging to Remote Hosts
One of the most useful
features of syslog is that syslogd can write to an Internet-domain socket connected to a socket on a remote
host. The local syslogd daemon opens a socket and writes messages to it. A remote syslogd daemon receives
the message. This socket is bound to Port 514/udp, which is reserved for syslog in /etc/services. Multiple
hosts can be specified, and remote hosts can be chained together. If one is chaining remote hosts together, one should be
aware that the logged message will list the system name of the sending system (which may or may not be the originating).
Here is an example of how
one would configure syslog for remote logging to the PageR NT/2K/XP/2003 system "admin" in the file /etc/syslog.conf:*.error <tab> @admin.hp.com # forward to PageR NT/2K/XP/2003
System-generated syslog messages
Many UNIX subsystems have
been converted to log their messages through syslog. One can now review most kernel-generated messages through the
syslog logfile. Kernel-generated messages are routed through the character-special device file /dev/klog.
Most messages generated by the kernel in bootup and operation are logged to syslog. In fact, most of the information
presented on the kernel bootup screens is logged. One can find the same information in the syslog log file that one
can find in the kernel message buffer--there is no longer any reason to use the dmesg(1m) command.
Many of the Internet services
(such as the inetd server and sendmail) have also been configured to log errors to syslog. The nettl
logging service used in network drivers and services is a different logging routine, but it announces its own startup
Since syslog operates over TCP/IP networks, networked peripherals can also use syslog services.
For example, HP printers that use the JetDirect network interface card can be configured to report error conditions to syslog.
This may be a convenient way to detect and report common printer errors.