Features and Benefits

Following are the main features and benefits of Solstice SyMON:

• Identifies hardware and software conditions quickly. Conditions range from major to minor. An example of a major condition is a CPU crash and an example of a minor condition is low swap space.
• Monitors hardware performance to detect incipient hardware failure; for example, soft read errors on a disk.

To give you this critical performance information, Solstice SyMON:

• Analyzes system performance in real time; when performance problems occur, the Solstice SyMON event system alerts you if desired
• Provides graphical displays of performance data

• Maintains a log file of Solstice SyMON-detected conditions for future analysis
• Provides a graphical display of all system components and their status
• Allows the launching of the SunVTS^(TM) diagnostic package to permit direct on-line diagnosis of hardware problems

Architecture

The Solstice SyMON internal application architecture is divided into two layers:

• Data Capture Layer (DCL)
• Management Application Program (MAP)

All monitored data originates in programs (or agents) of the DCL, which run continuously on the server. This data is sent to the MAP (which typically runs on a different machine in the network) where it is processed and displayed to the user.

The external architecture consists of the graphical user interface (GUI).

Internal Structure

The following section provides additional information on the Data Capture Layer (DCL) and Management Application Program (MAP) layers.

Data Capture Layer

The Data Capture Layer is comprised of three data-gathering agents and a daemon that controls them:

• Config Reader: This component reports system configuration data, such as the number of CPU boards, number and type of I/O boards. The Config Reader monitors and tracks changes to the configuration of the system, and the state of its components.
• Kernel Reader: This component extracts operating system data from the Solaris operating system kernel on a periodic basis. Some of the areas that are monitored and reported include:

• CPU usage
• Disk I/O
• Network I/O
• Memory usage
• Swap space usage
• Paging and swapping rates

• Log Scanner: This component monitors various system log files and searches for a user-specified list of regular expressions that are stored in the rule files. The Log Scanner generates "conditions." The GUI can also call on the Log Scanner to directly scan the log files. The scanner is reboot-recoverable so it can capture "panic" messages immediately following a system crash.
• symond daemon: This component starts when the server or the event host is booted. symond starts the agents, monitors their activity, and restarts them if they stop. It also provides a central contact point for any clients that want to connect to the agents.

Management Application Program

The Management Application Program is comprised of the:

• Graphical User Interface (GUI): The GUI receives and displays data from all agents in the DCL.
• Event Handler: The Event Handler receives and examines incoming data from the DCL and generates event messages when appropriate. The Event Handler notifies the user of a predefined condition (as defined in the event rule file) on the system being monitored.

The Event Handler monitors changes in DCL-supplied data and compares them to the conditions established in the event rule files. When the conditions of an event are met, the Event Handler generates an event message and, depending on the action specified in the rule, may highlight the troubled component in the GUI. It also notes the occurrence of the event in its log file. When the condition for an event no longer exists, it notes the termination time of that event in the log file depending on the action specified in the rule.

The GUI displays event messages from the Event Handler.

When an event is detected, in addition to adding it to the log file and displaying it in the GUI, specific user-defined actions may also occur. User-defined actions include sending an email message, firing an SNMP trap, or launching a script.

Solstice SyMON User Interface

Solstice SyMON provides the user with a GUI to control and interact with the tool. This GUI uses a CDE/Motif-based GUI or an OPEN LOOK GUI.

Solstice SyMON Launcher and Consoles

The Launcher appears when you start Solstice SyMON. The Launcher does the following (see Figure 1-1):

• Tests for available agents (ConfigTool, Kernel Reader, Log Scanner, and the Event Handler (as in the Launcher)
• Displays the various Solstice SyMON console icons and names; each icon displays an "unavailable" symbol until it receives data from the appropriate agent
• Shows the status of agents; red indicates that the GUI is unable to contact the agent, yellow indicates that the GUI has found the agent but is not yet receiving data

Figure 1-1

On the Launcher, there are seven consoles. Consoles allow you to view the data collected by Solstice SyMON. Table 1-1 describes the information presented by each console.

Table 1-1

Consoles	Displays
Event Viewer	Information about defined events (conditions) that occurred on the server
Log Viewer	Entries from the system log file
Physical View	Pictorial representation of the hardware of the server
Logical View	Hierarchical schematic view of the hardware components of the server

Table 1-1 (Continued)

Consoles	Displays
Kernel Data Catalog	Hierarchical schematic view of the monitored performance parameters
Process Viewer	Information about processes that are using system resources
On-Line Diagnostics	Connection to Sun VTS on-line diagnostics

Table 1-2 describes menu items that are typical in each console.

Table 1-2

Menu	Description
File	Gives you the option to close the window, exit the GUI, save a file or settings, or restore a file or settings
Edit	Offers different ways to manipulate windows, objects or text
View	Lists options for viewing data, or for displaying additional information about the data

Error Messages

Solstice SyMON generates error messages when a monitored operation is incorrect. Error messages are displayed in the footer of the active window.

Warning Messages

Warning messages are displayed in pop-up dialog boxes that appear above the console you are viewing. When a warning message is displayed, you must acknowledge it by selecting an action button (such as O. K. or Quit) in the dialog box.

Some warning messages state that a prerequisite for an operation has not been met. To acknowledge this message, click the Continue button before reattempting the operation.

Monitoring Features

This section provides an overview of the monitoring features of Solstice SyMON. There are three main groups of monitoring features:

• Hardware
• Software
• Alarms and events

Monitoring Hardware Status

Use Solstice SyMON to view the configuration of your system, monitor the status of system hardware components, and launch diagnostic tests. The three most frequently used consoles for monitoring hardware status are:

• Physical view
• Logical view
• On-line diagnostics

Physical View

The Physical View is a pictorial representation of the front and back of the server. The Physical View provides a picture of your system as it is actually configured. Information about each component and visual identification of a failed or troubled component is displayed.

Logical View

The Logical View is a companion view for the Physical View. While the Physical View displays the components of the system as physical entities, the Logical View displays the components in a schematic hierarchy.

On-line Diagnostics

Solstice SyMON can launch SunVTS to test and validate the CPUs, disks, and network connections of a server. Refer to the SunVTS 2.0 User's Guide for information about running SunVTS.

Monitoring System Software Performance

Solstice SyMON monitors software performance to isolate and identify system bottlenecks such as:

• Overloaded server
• Boards running hot

Solstice SyMON alerts the user to:

• Reconfigure software and hardware
• Plan for any system upgrades

Consoles for Monitoring Software Status

The three most frequently used consoles for monitoring software status are:

• Kernel Data Catalog
• Log Viewer
• Process Viewer

Kernel Data Catalog The Kernel Data Catalog provides a hierarchical diagram of the system performance parameters by category (CPU, memory, disk, network, and process), much like the Logical Viewer shows a hierarchical diagram of the hardware of the system.

System Meters The performance parameters that Solstice SyMON captures can be displayed as graphs in the System Meter console windows to show you the type and quantity of activities of the CPUs, disks, memory, and network interfaces of the server. See Table 1-3 for details.

Table 1-3

Component	Performance Parameters Captured
CPU	Performance across all CPUs on a server, or individual CPU performance; parameters that Solstice SyMON monitors are: use, system, total, wait time, idle times, context switches, interrupts, and so on
Memory	Memory available, page scan rate, total swap space, and so on
Disk	Number of active disks, projected fastest service time, number of bytes written, and average disk queue length, and so on
Network	Packet load and the number of packet errors that are occurring on the network interfaces of the server (overflow, underflow, CRC, frame, output, and input errors, as well as collisions)

Log Viewer The Log Viewer displays text written to the log file. It searches the log using a list of regular expressions. When a match is found, the corresponding log file entry is sent to the Log View.

Process Viewer The Process Viewer gathers and displays information about processes running on the server.

Monitoring Alarms and Events

Events are system conditions that require the attention of a system administrator. Alarms are used to alert the system administrator of an event. An alarm may be a message in the Event Viewer or a visual highlight in a console.

Event Handler

The Event Handler is the portion of Solstice SyMON that processes the performance data of the server. The Event Handler detects any event that violates the predefined performance parameters that one specified in a set of Event Rules. See Chapter 4 "Defining and Using Event Rules" for more information on the Event Handler.

The most frequently used console for monitoring events is the Event Viewer.

If the Event Handler detects problems with the system, the affected component is highlighted in red or yellow, depending on the severity of the event.

If a component is highlighted, all its related components are also highlighted. For example, if a CPU chip is highlighted, then the board on which it is mounted, and its chassis slot, are also highlighted. In addition, the Event Handler highlights the icon of the Launcher console.

Event Viewer The Event Viewer alerts you to any problems detected by the Event Handler. Each entry displayed in the Event Viewer window represents an event and gives you the following information:

• The rule number that caused the event to be generated
• The level of severity (red, yellow, or blue)
• The time and date the event was detected
• A message indicating the type of event that was captured
• The node where the event occurred
• The priority and severity of the event (predefined, user customizable)