Monitoring Hardware Status

You can use Solstice SyMON to view the configuration of a system, monitor the status of its hardware components, and launch diagnostic tests.

Using the Physical View

The Physical View is a pictorial representation of the front and back of the server. Use this console for information about any system components monitored by Solstice SyMON. When you launch the Physical View, a submenu with three viewing options appears:

• Front
• Back
• Front and the back

Displaying a Detailed View of a Component

When you view a system component, you can view its internal components as well, if available. For example, when you view a CPU or memory board mounted in a chassis, you can view a more detailed image of the CPU or memory board that includes the memory and processor modules.

When a more detailed sub-view is available, the Physical View cursor changes to a pointing hand.

· To Display a Detailed Picture of the Component

* Click the component with the "detail" pointer.

For example, if the pointer appears on a CPU panel in the chassis, and you click the CPU, Solstice SyMON displays another Physical View window with a detailed picture of the CPU or memory board (see Figure 3-1).

Figure 3-1

Show Information Option

The View Menu on the main Physical View and any sub-views of individual components include a Show Information option. You can use this option to view more specific information about each component of the system.

When you choose the Show Information option, Solstice SyMON displays a pop-up window. As you move the pointer to a component displayed on the Physical View, information about the component is displayed in the window (see Figure 3-2).

· To Use the Show Information Option

* Place the pointer on the View menu and select the Show Information option.

If the Show Information option is active and you place the pointer over a CPU displayed on the Physical View, Solstice SyMON displays critical information about the CPU, including: CPU type, status, cache-size, board number, port ID, processor ID, and model number.

You can display a particular Show Information display on your screen or save it to a file.

· To Save the Properties of the System Displayed in the Show Information Window

* Select the Save As option under the File menu of the Show Information window.

· To Close the Show Information Window

* Select the Close option on the Show Information File menu.

Figure 3-2

Close Options

· To Close a Physical View Using the File Pull-down Menu

* Select any of the following options:

• Close Self: Use this option to close this window only.
• Close Self and Descendants: Use this option to close this window and any more detailed component displays opened from this window.
• Close All: Use this option to close all Solstice SyMON Physical View windows.

· To Quickly Locate Troublesome Components

* Glance at the Physical View and trace events from the top level of the hierarchy down to the component that generated the event.

Using the Logical View

The Logical View is a companion view for the Physical View. While the Physical View displays the components of the system as physical entities, the Logical View displays the components in a schematic hierarchy (see Figure 3-3).

To show a more or less detailed listing of components, expand or contract branches of the hierarchy (for example CPU, I/O).

Figure 3-3

For information on how the Event Handler locates troublesome components, see the section, "To Quickly Locate Troublesome Components."

Expanding and Collapsing the Hierarchical Diagram

Solstice SyMON displays a two-state button to the left of each component in the hierarchy (see Figure 3-3). The two possible states are collapsed (+) and expanded (-).

• Collapsed(+)--Means that you can expand the view to display the next level of the hierarchy (only if it is not grayed out).
• Expanded (-)--Means that the display is already expanded.

Clicking the Collapsed(+) or Expanded (-) button expands or contracts the entire hierarchy below that point one level at a time.

· To Expand the Hierarchy

* Click the (+) button in the collapsed state.

The hierachy below that point is expanded. As you expand the hierarchy, you may need to resize the console to view the entire hierarchy.

· To Hide the Contents of the Hierarchy

* Click the button in the expanded (-) state.

Solstice SyMON hides the contents of the lower level of the hierarchy.

Show Information Option

The View Menu on the Logical View includes a Show Information option that is identical to the Physical View Show Information View Menu.

Use this option to view more specific information about each component of the system. Refer to the Show Information description in the section "Show Information Option" in this chapter.

Using the On-Line Diagnostics Console

The Diagnostics icon launches the Sun Validation Test Suite (SunVTS). SunVTS is a suite of diagnostics that tests and validates the configuration and functionality of most hardware devices of a server.

For more detailed information about SunVTS, refer to the SunVTS 2.0 User's Guide.

Monitoring System Software Performance

Use Solstice SyMON to monitor software performance and to identify system bottlenecks. Chapter 1, "Solstice SyMON Overview" covers how to isolate and identify system bottlenecks in the "Monitoring System Software Performance" section.

Using the Kernel Data Catalog to Build System Meters

The Kernel Data Catalog console displays the system performance parameters. These parameters are displayed as "instruments" within the Kernel Data Catalog. These parameters are arranged in a hierarchical diagram by performance category (CPU, memory, disk, network, and process), much like the Logical View shows a hierarchical diagram of the hardware of the system. You can choose whether the instruments associated with each hierarchy are displayed or hidden (see Figure 3-4).

Figure 3-4

When the status of a parameter results in an event, the Event Handler highlights the parameter on the Kernel Data Catalog. If a parameter is highlighted, its ancestors are also highlighted.

For information on how the Event Handler locates troublesome components, see the section "To Quickly Locate Troublesome Components." To expand or collapse the hierarchical diagram, see the section, "Expanding and Collapsing the Hierarchical Diagram" and the procedures "To Expand the Hierarchy" and "To Hide the Contents of the Hierarchy" earlier in this chapter.

Hiding and Showing Instruments

The View menu of the Kernel Data Catalog console includes an option that toggles between Hide Instruments and Show Instruments. Use this feature to hide or show instruments throughout the hierarchy.

Building System Meter Windows

The System Meter windows help to build a graphical display of system performance parameters.

Use the parameter instruments displayed in the Kernel Data Catalog to:

• Create a graph for one parameter in a System Meter
• Combine multiple parameters on the same graph in a System Meter
• Combine multiple graphs in the same System Meter

· To Create a System Meter

1. Ensure that the instrument for the parameter to be monitored is visible in the Kernel Data Catalog window.

2. Place the pointer over the desired parameter and click the left mouse button.

   Solstice SyMON creates a System Meter window with a graph of the selected parameter.

· To Add Another Parameter to an Existing Graph in a System Meter

* Place the pointer over the parameter you want to add and drag and drop it (using the middle or left mouse button, depending on your mouse setup).

Solstice SyMON adds the new parameter to the existing graph and legend (see Figure 3-5).

· To Add a New Graph to an Existing System Meter

* Place the pointer over the parameter you want to add and drag and drop it in the footer of the System Meter.

Solstice SyMON adds a second pane to the System Meter and draws the graph for the new parameter in the new pane.

---

Note - One graph has a slightly darker background than the other. The darker one is selected. Only one graph at a time per System Meter can be selected by clicking the pointer on the graph to select.

---

Figure 3-5

Customizing a System Meter

You can use the following options under the View menu to customize a System Meter (see Figure 3-6).

Figure 3-6

Graph Layout

The Graph Layout option controls the appearance of a System Meter that contains multiple graphs:

• Horizontal: Displays graphs horizontally, from left to right
• Vertical: Displays multiple graphs vertically, from top to bottom
• Matrix: Displays multiple graphs in a grid format

Graph Style

When you select Graph Style from the View pull-down menu, the Graph Style Dialog window appears (see Figure 3-7).

Figure 3-7

Modifying the System Meter Graphs

If you selected the "All Graphs" field in the style dialog window, the "All Graphs in Selected System Meter" field becomes inactive.

· To Modify the Selected Graph

1. Select the "Currently Selected Graph in Selected System Meter" field.

2. Use the bottom half of the Graph Style form to change the appearance of the graphs you selected.

Table 3-1 shows the graph attributes you can modify.

Table 3-1

Field	Option	Description
Use Defaults	Active/Inactive	Selects default settings for all fields at the bottom of the form
Y-Axis Scaling	Automatic	Makes Y-axis scaling changes to reflect
	Fixed	changing values (default) Makes Y-axis scaling constant, regardless of changes in values
Y-Axis Scaling	Linear	Makes Y-axis scaling linear (default)
	Log	Makes Y-axis scaling logarithmic, useful when a graph includes two parameters with vastly different values
Legend	Normal	Shows the legend (default)
	None	Omits the legend
	Long	Extended label for each variable. For example, cpu.cpu0.busy (%).
Graph Type	Area	Draws an area graph
	Line	Draws a line graph (default)

Table 3-1 (Continued)

Field	Option	Description
Y-Axis Minimum		Sets the minimum value for the Y-axis; used only if Y-Axis is "Fixed"
Legend Gravity	North NorthEast East	Top of the graph Right of the graph
	SouthEast South	Bottom of the graph (default)
	NorthWest West	Left of the graph
Legend Orientation	Vertical Horizontal	Displays the legend in a vertical format Displays the legend in a horizontal format (default)

Changing Selected Graphs

· To Change a Graph

1. Choose the Graph Styles option from the View menu of any System Meter.

2. Use the top section of the Graph Styles form to select the graphs that you want to change.

3. Make the desired changes to the controls on the bottom half of the Graph Styles form.

4. Click the Apply button.

· To Delete a Graph

1. Select a graph.

2. Select Delete from the SysMeter Edit menu.

   The selected graph is deleted.

· To Undo a Change in a Graph

* Select Undo from the SysMeter Edit menu.

· To Delete a Variable from a Graph

1. Place the pointer over the legend for the variable and click the LEFT mouse button.

   A dialog box asks, "Do you want to delete the following data set" with the following information contained within the dialog box:

• System Meter: Variable name
• Graph: Defines parameters being displayed
• Data Set: Lists the operation being monitored

2. Delete the variable by pressing the YES button.

Displaying the System Meter Graphs

You can move, rotate, scale, or zoom the contents of your graph to better display the information with the following procedures:

· To Move a Graph

1. Position the pointer within the graph.

2. Press and hold the Shift key on the keyboard and press and hold the middle mouse button.

3. Move the graph using the mouse.

· To Scale a Graph

1. Position the pointer within the selected graph.

2. Press and hold both the Control key then hold down the middle mouse button.

3. Move the mouse to scale the graph and release the mouse button to stop the graph motion.

· To Zoom a Graph

1. Press and hold down the Control key and the Shift key, and the left mouse button at one corner of the graph.

2. Drag the mouse to form a rectangle around the area you want to magnify, while holding down the middle mouse button.

3. Release the mouse button to define the rectangle and to complete the zooming action.

---

Note - If you release the Control or Shift keys too early, the graph starts a drag-and-drop operation instead of zooming.

---

· To Restore a Graph to the Default Setting

* Position the cursor within the desired graph and type r.

Saving and Redisplaying System Meter Configurations

You can save and recall System Meter layouts and configurations you have defined. For example, you may have a specific set of CPU monitoring graphs that you would like to view every time your run Solstice SyMON.

· To Save the System Meter Configuration into a New File

1. Choose the Save As option from the View menu.

   The software displays the File Selection Dialog box.

2. Select the directory and file name from the Directories and Files panes or enter a new file name in the Selection pane.

· To Save an Altered System Meter Configuration

* Choose the Save option from the File menu.

Solstice SyMON writes the SysMeter contents to the existing file and displays the name of the saved file in the footer of the System Meter window.

· To recall a saved System Meter

1. Choose the Open option from the File menu.

   The software displays the File Selection Dialog box.

2. Select the file name from the Directories and Files panes or enter the file name in the Selection panel.

---

Note - You can save or restore sets of System Meters by using the File pull-down menu in the Kernel Data Catalog.

---

Using the Log Viewer

Use this console to search the system log file for entries that occur at a specific time or for entries that contain a specific keyword, or word pattern. Opening the Log Viewer console displays the Log File Viewer window as shown in Figure 3-8.

---

Note - A new Log File Viewer window is opened each time you select the Log Viewer console button on the Launcher. This allows different criteria to be monitored at the same time.

---

Figure 3-8

The View pull-down menu displays the following:

• Log File Status: Describes the Log Scanner status
• Show Search Criteria Summary: Toggle button that displays or hides the Search Criteria Summary
• Display Oldest/Newest Message First: Defines the sort criteria displayed in the Search Result window; this option can be toggled to display either the Oldest message first or the Newest message first

Searching for Log Entries

Initially, the Search Results window will be empty until you specify the Search Criteria and initiate the search. You can specify your own Search Criteria, retrieve and use a stored Search Criteria, or retrieve and modify a stored Search Criteria as defined further in this section.

Log entries retrieved by a search includes the items listed in Table 3-2.

Table 3-2

Search for	Description
Log File	Name of the log file viewed
Search for Expression	User-defined keyword, string, or regular expression
Messages From	Start date and start time of search
Messages To	End date and end time of search
Periodic Query	interval (in minutes) between repeated searches
Maximum Number of Matches	Only this number of messages is to be reported
Search Order	Oldest ( or newest) message searched first

If Solstice SyMON locates any entries in the selected log file that meet the search criteria, the entries are displayed in the Search Results pane. To change the order of the information you gathered in your search, proceed as follows:

· To Sort the Results of the Search

* Select either the Display Oldest Message First or the Newest Message First toggle button in the View menu to display messages in the desired search order.

· To Change the Search Criteria

* Select Change Search Criteria from the Log File Viewer window.

· To Initiate a Search

* Select Start Search from the Log File Viewer window.

· To Stop a Search

* Select Stop Search from the Log File Viewer window.

---

Note - Due to the client-server structure of Solstice SyMON, the search may appear to continue for several seconds. This will be more evident when the server is fully loaded.

---

· To Save the Search Results

* Select Save Search Results from the File pull-down menu.

· To Save the Search Criteria

* Select the Save Search Criteria option in the File pull-down menu.

Figure 3-9

Searching for Log File Entries

You can define your search criteria for specific log file entries by selecting the Change Search Criteria button in the Log File Viewer window.

· To Specify the Start Date, Start Time, End Date, and End Time

1. Move the pointer to the "From" field text field.

   Enter the time in 24-hour format (see Figure 3-10). The "From" time defaults to when the log was created.

2. Move the pointer to the "To" field text field.

   Enter the time and date field. The "To" time defaults to the current time.

· To Search for Log File Entries that Contain a Specific Expression or Keyword

1. Move to the Search for Expression field.

2. Enter the expression or keyword.

You can also perform more complex searches by specifying a grep-like regular expression instead of a keyword. For regular expression syntax, refer to the grep man page.

Figure 3-10

The Worksheet entries can be modified, or new entries can be entered into the desired search parameters, to include:

• Searches for a specific keyword or expression
• Events that occur during a specific time and contain a specific keyword
• Periodic monitoring of specific events
• Maximum number of matches of these search criteria to be reported
• Listing the messages in ascending or descending order of occurrence

Using the Process Viewer

The Process Viewer displays information on processes running on the server (see Figure 3-11). Each entry displayed on this console represents a process with some of the following data:

• When the process was started
• Who started the process
• How much CPU time was used by the process
• Current process status

Figure 3-11

Monitoring Resource-Intensive Processes

Resource-intensive processes such as compiles, simulations, and complex database manipulations can slow server performance. The Process Viewer console gives you detailed information about resource-consuming processes that run on your server. This information can help you identify resource hogs that can contribute to service problems.

Processes that are inactive or are consuming little CPU time are not reported. This feature eliminates the need to search through dozens of relatively idle processes to view data on resource-intensive processes. In fact, a process may appear and disappear from this console as its resource consumption changes.

Table 3-3 describes the information displayed in each Process Viewer entry:

Table 3-3

Field	Description
pid	Process ID
command	Name of program
uname	User name
uid	User ID
nice	Determines when the process can voluntarily lower its priority. As the nice value increases, the process is more willing to give up CPU (value defaults to 20). For more information, see the nice man page.
status	Lists the current status of the process: - Sleeping: Not generating any activity - Running: Generating activity - Runnable: Waiting for I/O - Trace: A debugger is running
user (%)	Amount of CPU time that the process is using
sys (%)	Amount of CPU time the kernel is using to perform tasks

Table 3-3 (Continued)

Field	Description
Total (%)	Percentage of total CPU time used over the entire life of the process
Total CPU (%)	The combined seconds and percentage of CPU time used by the system, user, and short-lived processes
Start up time	Time the process started; if the process has been running for more than 24 hours, it displays the date the process started

Customizing the Process Viewer Display

You can customize the Process Viewer window to suit your specific display requirements.

· To Organize the Process Viewer Window

* Use the options under the View menu to organize the Process Viewer.

Sorting The View pull-down menu options are:

• Sort: The menu displays a list of the fields (columns) for each process view. You can choose one of these fields as the basis for sorting the process views. You can also click on the column's title bar to change the sort order.
• Ascending/Descending Sort Order: The current direction (ascending or descending) of the sort is displayed; if you click the option, it will toggle to the opposite direction.

Adjusting Column Widths

You can also resize the columns.

· To Resize the Column

1. Place the pointer in the column that you want to resize.

2. Move the pointer to the left or right margin of the column.

   The pointer changes from a cross hair to an arrow with a line to right

3. Press and hold the left mouse button.

4. Drag the margin to the desired spot, and release the mouse button.

Advanced Features

You can set the order in which columns are displayed, their default width, and the default sort column through the common.tcl file, which is read when Solstice SyMON starts. See the "Customizing the Appearance of the Event View and Process View" section at the end of this chapter.

Monitoring Alarms and Events

The Event Handler monitors data from the server and generates events when anomalous conditions occur on your server. The condition, and their corresponding alarms, are described by a set of event rules. See Chapter 4, "Understanding and Using Event Rules".

Using the Event Viewer

If the Event Handler senses an event, it adds a description of the event to the Event Log (see Figure 3-12). The Event Log is displayed via the Event Viewer.

Figure 3-12

Table 3-3 displays the information found in each event log entry. Some information is optional.

Table 3-4

Field	Description
ID	Sequential number assigned to the event
Lvl	Level of events: yellow (caution), red (danger)
Opened	Time the event occurred
Message	Event message (failure detected, for example)
Node	Where the event was detected
Pri	Priority of the event rule
Sev	Severity level of the event
Acked	Time a user acknowledged the event
Ack By	User ID of person acknowledging the event
Rule ID	Number of the rule that caused the event

Two displays are available from the Event Viewer:

• Open Events: Displays only currently open events. This is the view you see when you select the Event Viewer icon from the launcher.
• All Events: Displays all events (even if closed), until they are explicitly deleted.

When an event is open, the Event Handler highlights the event in the Logical View and Physical View or the Kernel Data Catalog until the cause of the event is corrected.

When the condition that caused an event ends, the description of the event is removed from the Open Event view and any associated highlights in the consoles are also removed. The Event Handler notes the closing of the event in the event log.

Acknowledging and Deleting Events

When you acknowledge an event, it has no effect on the event condition or alarms. However, it informs all Solstice SyMON users that the event has been noticed.

· To Acknowledge an Event

1. Select the event to be acknowledged.

2. Select Acknowledge from the Edit pull-down menu.

   The Event Handler enters the time and date in the Acked column and your User ID in the Ack By column.

---

Note - If you acknowledge or delete an event, the change appears in all Solstice SyMON GUIs.

---

After the event is closed, it remains in the All Events view until a Solstice SyMON user deletes it.

· To delete a closed event

1. Select the All Events from the View pull-down menu.

2. Select the event to be deleted.

3. Select Delete from the Edit pull-down menu.

---

Note - You cannot delete an Open event from the event list.

---

Customizing the Event Viewer

Like the Process Viewer Display, you can customize the Event Viewer window to suit your specific display requirements.

Sorting Event Data To use the options under the View menu to sort Event Data, see the section "Sorting" earlier in this chapter.

Adjusting Column Widths To resize the columns, see the procedure, "To Resize the Column" earlier in this chapter.

Customizing the Event Viewer and Process Viewer

The appearance of the Event Viewer and Process Viewer tabular displays of the Event View and Process Viewer are controlled by the file ~/.symon/lib/tcl/common.tcl. This file is read and interpreted when the Solstice SyMON GUI is started. You can customize the appearance of these consoles by editing this file.

In this file, three major Tcl variables are set: symon_process_columns, event_display_open_columns, and event_display_all_columns. These variables control the data that appears in the display; the order in which they appear; and the default column width for each column.

Within each of these variables is a line for each data item. The order in which these data item lines appear in the Tcl variable is the order in which their respective columns appear in the displays.

A typical line looks like this:

{"tpcnt"  "  Total (%)  "    "+"}

The first string is the Solstice SyMON internal name for the data item. The second string is the title for the column in which it is displayed. The total length of this string (including spaces) controls the default width of the column. The third string (which can only be "+" or "-") controls the default sort order of the column (+ for ascending, - for descending). Some lines may have a fourth string with a format specifier such as %s or %d; these are unused and may be omitted.

Associated with each of the appearance-controlling Tcl variables are two other variables: event_display_all_columns_alternate and event_display_all_columns_sort_node. The _sort_node variable takes a single argument: the name of the data item used to sort the display when it initially appears.

The _alternate variable is used to hold control lines for data items that do not appear in the display.