About Monitoring Performance

While your computer is running, counters in the operating system are incremented to keep track of various system activities. System activities that are tracked are:

• Central processing unit (CPU) utilization
• Buffer usage
• Disk and tape input/output (I/O) activity
• Terminal device activity
• System call activity
• Context switching
• File access
• Queue activity
• Kernel tables
• Interprocess communication
• Paging
• Free memory and swap space
• Kernel Memory Allocation (KMA)

The following sections describe tools and commands that help you monitor performance.

The sar Command

Use the sar command to:

• Organize and view data about system activity
• Access system activity data on a special request basis
• Generate automatic reports to measure and monitor system performance, and special request reports to pinpoint specific performance problems. "Automatic Collection of System Activity Data" on page 133 describes these tools.

The vmstat Command

The vmstat command reports virtual memory statistics and shows CPU load, paging, number of context switches, device interrupts, and system calls.

The following example shows the vmstat display of statistics gathered at five-second intervals.

example% vmstat 5 procs memory page disk faults cpu r b w swap free re mf pi po fr de sr f0 s3 -- -- in sy cs us sy id 0 0 8 28312 668 0 9 2 0 1 0 0 0 1 0 0 10 61 82 1 2 97 0 0 3 31940 248 0 10 20 0 26 0 27 0 4 0 0 53 189 191 6 6 88 0 0 3 32080 288 3 19 49 6 26 0 15 0 9 0 0 75 415 277 6 15 79 0 0 3 32080 256 0 26 20 6 21 0 12 1 6 0 0 163 110 138 1 3 96 0 1 3 32060 256 3 45 52 28 61 0 27 5 12 0 0 195 191 223 7 11 82 0 0 3 32056 260 0 1 0 0 0 0 0 0 0 0 0 4 52 84 0 1 99

The fields in the vmstat report have the following meanings:

procs reports the following states:

• r..The number of kernel threads in the dispatch queue
• b..Blocked kernel threads waiting for resources
• w..Swapped out LWPs waiting for processing resources to finish

memory reports on usage of real and virtual memory:

• swap Available swap space
• free Size of the free list

page reports on page faults and paging activity, in units per second:

• re..Pages reclaimed
• mf..Minor and major faults
• pi..Kbytes paged in
• po..Kbytes paged out
• fr..Kbytes freed
• de..Anticipated memory needed by recently swapped-in processes
• sr..Pages scanned by page daemon (not currently used)

If sr does not equal zero, the page daemon has been running.

disk reports the number of disk operations per second. This field can show data on up to four disks.

faults reports the trap/interrupt rates (per second):

• in..Interrupts per second
• sy..System calls per second
• cs..CPU context switch rate

cpu reports on the use of CPU time:

• us..User time
• sy..System time
• id..Idle time

The vmstat command can also display statistics on swapping, cache flushing, and interrupts.

System Events Run vmstat -s to show the total of various system events that have taken place since the system was last booted.

Swapping Run vmstat -S to show swapping statistics in addition to paging statistics. The additional fields are:

• si..Average number of LWPs swapped in per second
• so..Number of whole processes swapped out

---

Note - The vmstat command truncates the output of both of these fields. Use the sar command to display a more accurate accounting of swap statistics.

---

Cache Flushing Run vmstat -c to show cache flushing statistics for a virtual cache. It shows the total number of cache flushes since the last boot. The cache types are:

• usr User
• ctx Context
• rgn Region
• seg Segment
• pag Page
• par Partial-page

Interrupts Run vmstat -i to show interrupts per device.

example% vmstat -i interrupt total rate --------------------------------- clock 104638405 100 esp0 2895003 2 fdc0 0 0 --------------------------------- Total 107533408 102

The iostat Command

The iostat command reports statistics about disk input and output, and produces measures of throughput, utilization, queue lengths, transaction rates, and service time.

The following example shows disk statistics gathered every five seconds:

example% iostat 5 tty fd0 sd3 cpu tin tout bps tps serv bps tps serv us sy wt id 0 1 0 0 0 1 0 5640 0 1 0 98 0 10 0 0 0 0 0 0 0 1 0 99 0 10 0 0 0 0 0 0 0 1 0 99 0 10 0 0 0 27 3 319 0 4 9 88 0 10 0 0 0 2 0 5061 0 0 0 99 0 10 0 0 0 0 0 0 0 0 0 100 0 10 0 0 0 0 0 0 0 0 0 100 0 10 0 0 0 0 0 0 0 0 0 100 0 10 0 0 0 0 0 0 0 0 0 100

The first line of output shows the statistics since the last boot. Each subsequent line shows the interval statistics. The default is to show statistics for the terminal (tty), disks (fd and sd), and CPU (cpu).

For each terminal, iostat displays:

• tin Number of characters in the terminal input queue
• tout Number of characters in the terminal output queue

For each disk, iostat displays the following information:

• bps Blocks per second
• tps Transactions per second
• serv Average service time, in milliseconds

For the CPU, iostat displays the CPU time spent in the following modes:

• us..In user mode
• sy..In system mode
• wt..Waiting for I/O
• id..Idle

Run iostat -xtc to get extended disk statistics.

example% iostat -xtc extended disk statistics tty cpu disk r/s w/s Kr/s Kw/s wait actv svc_t %w %b tin tout us sy wt id sd0 0.2 1.7 1.0 9.7 0.0 0.1 39.8 0 3 0 9 1 6 9 85 sd1 0.5 2.5 10.6 21.0 0.0 0.1 26.6 0 5 sd2 0.0 0.2 0.1 0.0 0.0 0.0 157.7 0 0

Each disk has a line of output:

• r/s Reads per second
• w/s Writes per second
• Kr/s Kbytes read per second
• Kw/s Kbytes written per second
• wait Average number of transactions waiting for service (queue length)
• actv Average number of transactions actively being serviced
• svc_t Average service time, in milliseconds
• %w..Percentage of time the queue is not empty
• %b..Percentage of time the disk is busy

The df Command

The df command shows the amount of free disk space on each mounted disk. The usable disk space reported by df reflects only 90% of full capacity, as the reporting statistics leave a 10% head room above the total available space. This head room normally stays empty for better performance. The percentage of disk space actually reported by df is used space divided by usable space. If the file system is above 90% capacity, transfer files to a disk that is not as full by using cp, or to a tape by using tar or cpio; or remove the files.

Use the df -k command to display file system information in Kbytes. The following information is given:

• kbytes...Total size of usable space in the file system
• used....Amount of space used
• avail...Amount of space available for use
• capacity Amount of space used, as a percent of the total capacity
• mounted on Mount point

example% df -k filesystem kbytes used avail capacity mounted on /dev/dsk/c0t3d0s0 17269 11099 4450 71% / /dev/dsk/c0t3d0s6 136045 79818 42627 65% /usr /proc 0 0 0 0 /proc swap 40424 0 40416 0 /tmp

The profil Command

profil uses CPU statistics to show the amount of time that a program uses. You can analyze a program and identify the functions that consume a high percentage of CPU time. See the reference page for profil(2) for more information.

Performance Meter

The Performance Meter (perfmeter) offers a way to view and monitor various performance parameters for your system, or for other systems on the network. The performance meters graphically display the status of the following system parameters:

• CPU
• Ethernet packets per second
• Paging activity in pages per second
• Jobs swapped per second
• Number of device interrupts per second
• Disk traffic in transfers per second
• Number of context switches per second
• Average number of runnable processes over the last minute
• Collisions per second detected on the Ethernet
• Errors per second on receiving packets

The main advantage of the performance meters is that they give you an almost immediate feedback on system performance. You can execute a command or a program, and watch the graph to see if CPU usage makes a big jump. Or you can display the load of several hosts on the network, and choose the one with the smallest load on which to run a CPU-intensive program.

Figure 8-1 shows a performance meter displaying CPU activity, in percent. The curve moves from right to left, showing the CPU load on the system.

Figure 8-1

Automatic Collection of System Activity Data

Three commands are involved in automatic system activity data collection: sadc, sa1, and sa2.

The sadc data collection utility periodically collects data on system activity and saves it in a file in binary format--one file for each 24-hour period. You can set up sadc to run periodically (usually once each hour), and whenever the system boots to multiuser mode. The data files are placed in the directory /usr/adm/sa. Each file is named sadd, where dd is the current date. The format of the command is as follows:

/usr/lib/sa/sadc [t n] [ofile]

The command samples n times with an interval of t seconds (t should be greater than 5 seconds) between samples. It then writes, in binary format, to the file ofile, or to standard output. If t and n are omitted, a special file is written once.

Running sadc When Booting

The sadc command should be run at system boot time in order to record the statistics from when the counters are reset to zero. To make sure that sadc is run at boot time, the /etc/init.d/perf file must contain a command line that writes a record to the daily data file.

The command entry has the following format:

su sys -c "/usr/lib/sa/sadc /usr/adm/sa/sa'date +5d'"

Running sadc Periodically with sa1

To generate periodic records, you need to run sadc regularly. The simplest way to do this is by putting a line into the /var/spool/cron/sys file, which calls the shell script, sa1. This script invokes sadc and writes to the daily data files, /var/adm/sa/sadd. It has the following format:

/usr/lib/sa/sa1 [t n]

The arguments t and n cause records to be written n times at an interval of t seconds. If these arguments are omitted, the records are written only one time.

Producing Reports With sa2

There is another shell script, sa2, which produces reports rather than binary data files. The sa2 command invokes the sar command and writes the ASCII output to a report file.

Collecting System Activity Data With sar

The sar command can be used either to gather system activity data itself or to report what has been collected in the daily activity files created by sadc.

The sar command has the following formats:

sar [-aAbcdgkmpqruvwy] [-o file] t [n]
sar [-aAbcdgkmpqruvwy] [-s time] [-e time] [-i sec] [-f file]

The sar command below samples cumulative activity counters in the operating system every t seconds, n times. (t should be 5 seconds or greater; otherwise, the command itself may affect the sample.) You must specify a time interval between which to take the samples; otherwise, the command operates according to the second format. The default value of n is 1. The following example takes two samples separated by 10 seconds. If the -o option is specified, samples are saved in file in binary format.

example% sar -u 10 2

Other important information about the sar command:

• With no sampling interval or number of samples specified, sar extracts data from a previously recorded file, either the one specified by the -f option or, by default, the standard daily activity file, /var/adm/sa/sadd, for the most recent day.
• The -s and -e options define the starting and ending times for the report. Starting and ending times are of the form hh[:mm[:ss]] (where h, m, and s represent hours, minutes, and seconds).
• The -i option specifies, in seconds, the intervals between record selection. If the -i option is not included, all intervals found in the daily activity file are reported.

Table 8-1 lists the sar options and their actions.

Table 8-1 sar

Option	Actions
-a	Checks file access operations
-b	Checks buffer activity
-c	Checks system calls
-d	Checks activity for each block device
-g	Checks page-out and memory freeing
-k	Checks kernel memory allocation
-m	Checks interprocess communication
-p	Checks swap and dispatch activity
-q	Checks queue activity
-r	Checks unused memory
-u	Checks CPU utilization
-v	Checks system table status
-w	Checks swapping and switching volume
-y	Checks terminal activity
-A	Reports overall system performance (same as entering all options)

If no option is used, it is equivalent to calling the command with the -u option.

Checking File Access With sar -a

The sar -a option reports on the use of file-access operations. The operating system routines reported are as follows:

iget/s	The number of requests made for inodes that were not in the directory name lookup cache (dnlc).
namei/s	This is the number of file-system path searches per second. If namei does not find a directory name in the dnlc, it calls iget to get the inode for either a file or directory. Hence, most igets are the result of dnlc misses.
dirbk/s	This is the number of directory block reads issued per second.

The following is an example of sar -a output. It illustrates a one-minute sampling interval.

Solaris mysys Solaris 2.3 sun4c 08/22/93 14:28:12 iget/s namei/s dirbk/s 14:29:12 0 2 1 14:30:12 0 4 1 14:31:12 0 3 1 Average 0 3 1

The larger the values reported, the more time the kernel is spending to access user files. The amount of time reflects how heavily programs and applications are using the file systems. The -a option is helpful for viewing how disk-dependent an application is.

Checking Buffer Activity with sar -b

The buffer cache is used to cache metadata, which includes inodes, cylinder group blocks, and indirect blocks. The -b option reports on the following buffer activities:

bread/s	Average number of reads per second submitted to the buffer cache from the disk.
lread/s	Average number of logical reads per second from the buffer cache.
%rcache	Fraction of logical reads found in the buffer cache
bwrit/s	Average number of physical blocks (512 blocks) written from the buffer cache to disk, per second.
lwrite/s	Average number of logical writes to the buffer cache, per second.
%wcache	Fraction of logical writes found in the buffer cache
pread/s	Average number of physical reads, per second, using character device interfaces.
pwrit/s	Average number of physical write requests, per second, using character device interfaces.

The most important entries are the cache hit ratios %rcache and %wcache, which measure the effectiveness of system buffering. If %rcache falls below 90, or if %wcache falls below 65, it may be possible to improve performance by increasing the buffer space.

The following is an example of sar -b output:

14:28:12 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s 14:29:12 0 14 100 6 17 67 0 0 14:30:12 0 12 99 6 16 65 0 0 14:31:12 0 12 100 6 16 65 0 0 Average 0 12 100 6 16 66 0 0

This example shows that the buffers are not causing any slowdowns, because all the data is within acceptable limits.

Checking System Calls With sar -c

The -c option reports on system calls in the following categories:

scall/s	All types of system calls per second (generally about 30 per second on a busy four- to six-user system).
sread/s	read system calls per second.
swrit/s	write system calls per second.
fork/s	fork system calls per second (about 0.5 per second on a four- to six-user system); this number will increase if shell scripts are running.
exec/d	exec system calls per second; if exec/s divided by
rchar/s	Characters (bytes) transferred by read system calls per second.
wchar/s	Characters (bytes) transferred by write system calls per second.

Typically, reads and writes account for about half of the total system calls, although the percentage varies greatly with the activities that are being performed by the system.

The following is an example of sar -c output:

Solaris mysys Solaris 2.3 sun4c 08/22/93 14:28:12 scall/s sread/s swrit/s fork/s exec/s rchar/s wchar/s 14:29:12 17 2 2 0.28 0.28 2527 1542 14:30:12 25 2 1 0.50 0.47 1624 295 14:31:12 21 2 2 0.35 0.35 1812 703 Average 21 2 2 0.38 0.37 1987 847

Checking Disk Activity With sar -d

The sar -d option reports the activities of disk devices.

device	Name of the disk device being monitored.
%busy	Percentage of time the device spent servicing a transfer request.
avque	The sum of the average wait time plus the average service time.
r+w/s	Number of read and write transfers to the device per second.
blks/s	Number of 512-byte blocks transferred to the device per second.
avwait	Average time, in milliseconds, that transfer requests wait idly in the queue (measured only when the queue is occupied).
avserv	Average time, in milliseconds, for a transfer request to be completed by the device (for disks, this includes seek, rotational latency, and data transfer times).

The following two examples illustrate the sar -d output. The first example is from a computer with a non-SCSI (Small Computer System Interface, pronounced "scuzzy") integral disk; that is, a disk that does not use a SCSI interface. This example illustrates data being transferred from a hard disk (hdsk-0) to the floppy disk (fdsk-0):

Solaris mysys Solaris 2.3 sun4c 8/11/93 13:46:28 device %busy avque r+w/s blks/s avwait avserv 13:46:58 hdsk-0 6 1.6 3 5 13.8 23.7 fdsk-0 93 2.1 2 4 467.8 444.0 13:47:28 hdsk-0 13 1.3 4 8 10.8 32.3 fdsk-0 100 3.1 2 5 857.4 404.1 13:47:58 hdsk-0 17 .7 2 41 .6 48.1 fdsk-0 100 4.4 2 6 1451.9 406.5 Average hdsk-0 12 1.2 3 18 8.4 34.7 fdsk-0 98 3.2 2 5 925.7 418.2

The second example is from a computer with SCSI integral disks; that is, disks that use a SCSI interface. The example illustrates data being transferred from one SCSI hard disk (sd00-0) to another SCSI integral disk (sd00-1):

Solaris mysys Solaris 2.3 sun4c 8/11/93 14:16:24 device %busy avque r+w/s blks/s avwait avserv 14:16:52 sd00-0 2 1.0 1 3 0.0 17.9 sd00-1 6 1.1 3 5 2.0 23.9 14:17:21 sd00-0 2 1.0 1 2 0.0 19.6 sd00-1 6 1.1 3 5 0.2 24.3 14:17:48 sd00-0 3 1.0 1 3 0.3 18.3 sd00-1 7 1.1 3 5 1.3 25.4 14:18:15 sd00-0 3 1.0 1 3 0.0 17.2 sd00-1 5 1.0 2 5 0.0 21.6 Average sd00-0 2 1.0 1 3 0.1 18.2 sd00-1 6 1.0 3 5 0.9 23.0

Note that queue lengths and wait times are measured when there is something in the queue. If %busy is small, large queues and service times probably represent the periodic efforts by the system to ensure that altered blocks are written to the disk in a timely fashion.

Checking Page-out and Memory With sar -g

The sar -g option reports page-out and memory freeing activities (in averages) as follows:

pgout/s	The number of page-out requests per second.
ppgout/s	The actual number of pages that are paged-out, per second. (A single page-out request may involve paging-out multiple pages.)
pgfree/s	The number of pages, per second, that are placed on the free list.
pgscan/s	The number of pages, per second, scanned by the page daemon. If this value is high, the page daemon is spending a lot of time checking for free memory. This implies that more memory may be needed.
%ufs_ipf	The percentage of ufs inodes taken off the free list by iget that had reusable pages associated with them. These pages are flushed and cannot be reclaimed by processes. Thus, this is the percentage of igets with page flushes. A high value indicates that the free list of inodes is page-bound and the number of ufs inodes may need to be increased.

The following is an example of sar -g output:

Solaris mysys Solaris 2.3 sun4c 08/22/93 14:28:12 pgout/s ppgout/s pgfree/s pgscan/s %ufs_ipf 15:29:13 0.00 0.00 0.35 8.18 0.00 16:29:12 1.20 2.20 3.35 3.40 0.00

sar -g is a good indicator of whether more memory may be needed. Use the ps -elf command to show the number of cycles used by the page daemon. A high number of cycles, combined with high values for pgfree/s and pgscan/s indicates a memory shortage.

sar -g also shows whether inodes are being recycled too quickly, causing a loss of reusable pages.

Checking Kernel Memory Allocation With sar -k

The sar -k option reports on the following activities of the Kernel Memory Allocator (KMA):

sml_mem	The amount of memory, in bytes, that the KMA has available in the small memory request pool (a small request is less than 256 bytes).
alloc	The amount of memory, in bytes, that the KMA has allocated from its small memory request pool to small memory requests.
fail	The number of requests for small amounts of memory that failed.
lg_mem	The amount of memory, in bytes, that the KMA has available in the large memory request pool (a large request is from 512 bytes to 4 Kbytes).
alloc	The amount of memory, in bytes, that the KMA has allocated from its large memory request pool to large memory requests.
fail	The number of failed requests for large amounts of memory.
ovsz_alloc	The amount of memory allocated for oversized requests (those greater than 4 Kbytes); these requests are satisfied by the page allocator--thus, there is no pool.
fail	The number of failed requests for oversized amounts of memory.

The KMA allows a kernel subsystem to allocate and free memory as needed. Rather than statically allocating the maximum amount of memory it is expected to require under peak load, the KMA divides requests for memory into three categories: small (less than 256 bytes), large (512 to 4 Kbytes), and oversized (greater than 4 Kbytes). It keeps two pools of memory to satisfy small and large requests. The oversized requests are satisfied by allocating memory from the system page allocator.

If you are investigating a system that is being used to write drivers or STREAMS that use KMA resources, then sar -k will likely prove useful. Otherwise, you will probably not need the information it provides. Any driver or module that uses KMA resources, but does not specifically return the resources before it exits, can create a memory leak. A memory leak causes the

amount of memory allocated by KMA to increase over time. Thus, if the alloc fields of sar -k increase steadily over time, there may be a memory leak. Another indication of a memory leak is failed requests. If this occurs, then it is likely that a memory leak has caused KMA to be unable to reserve and allocate memory.

If it appears that a memory leak has occurred, you should check any drivers or STREAMS that may have requested memory from KMA and not returned it.

The following is an example of sar -k output:

Solaris mysys Solaris 2.3 sun4c 08/22/93 14:28:12 sml_mem alloc fail lg_mem alloc fail ovsz_alloc fail 14:29:12 95232 73472 0 311296 198656 0 180224 0 14:30:12 95232 75120 0 311296 198656 0 180224 0 14:31:12 95232 73600 0 311296 197632 0 180224 0 Average 95232 74064 0 311296 198314 0 180224 0

Checking Interprocess Communication With sar -m

The sar -m option reports interprocess communication activities.

msg/s	The number of message operations (sends and receives) per second.
sema/s	The number of semaphore operations per second.

An example of sar -m output follows:

Solaris mysys 2.0 sun4c 08/22/93 14:28:12 msg/s sema/s 14:29:12 0.00 0.00 14:30:12 0.00 0.00 14:31:12 0.00 0.00 Average 0.00 0.00

These figures will usually be zero (0.00), unless you are running applications that use messages or semaphores.

Checking Page-in Activity With sar -p

The sar -p option reports page-in activity which includes protection and translation faults. It reports the following statistics:

atch/s	The number of page faults, per second, that are satisfied by reclaiming a page currently in memory (attaches per second). Instances of this include reclaiming an invalid page from the free list and sharing a page of text currently being used by another process (for example, two or more processes accessing the same program text).
pgin/s	The number of times, per second, that file systems receive page-in requests.
ppgin/s	The number of pages paged in, per second. A single page-in request, such as a soft-lock request (see slock/s), or a large block size, may involve paging-in multiple pages.
pflt/s	The number of page faults from protection errors. Instances of protection faults are illegal access to a page and "copy-on-writes." Generally, this number consists primarily of "copy-on-writes."
vflt/s	The number of address translation page faults, per second. These are known as validity faults, and occur when a valid process table entry does not exist for a given virtual address.
slock/s	The number of faults, per second, caused by software lock requests requiring physical I/O. An example of the occurrence of a soft-lock request is the transfer of data from a disk to memory. The system locks the page that is to receive the data, so that it cannot be claimed and used by another process.

The following is an example of sar -p output:

Solaris mysys Solaris 2.3 sun4c 08/22/93 14:28:12 atch/s pgin/s ppgin/s pflt/s vflt/s slock/s 14:29:12 1.17 12.87 12.87 5.67 11.28 1.15 14:30:12 1.67 7.08 7.08 9.12 6.33 0.67 14:31:12 1.37 12.48 12.48 6.83 10.78 1.03 Average 1.40 10.81 10.81 7.21 9.46 0.95

Checking Queue Activity With sar -q

The sar -q option reports the average queue length while the queue is occupied, and the percentage of time that the queue is occupied.

runq-sz	The number of kernel threads in memory waiting for a CPU to run. Typically, this value should be less than 2. Consistently higher values mean that the system may be CPU-bound.
%runocc	The percentage of time the dispatch queues are occupied.
swpq-sz	The average number of swapped out LWPs.
%swpocc	The percentage of time LWPs are swapped out.

An example of sar -q output follows:

Solaris mysys Solaris 2.3 sun4c 08/22/93 14:28:12 runq-sz %runocc swpq-sz %swpocc 14:29:12 1.2 53 1 100 14:30:12 1.3 38 14:31:12 1.1 37 Average 1.2 43

---

Note - The number of LWPs swapped out may greater than zero even if the system has an abundance of free memory. This happens when a sleeping LWP is swapped out and has not been awakened (for example, a process or LWP sleeping, waiting for the keyboard or mouse input).

---

If %runocc is high (greater than 90 percent) and runq-sz is greater than 2, the CPU is heavily loaded and response is degraded. In this case, additional CPU capacity may be required to obtain acceptable system response.

Checking Unused Memory With sar -r

The -r option records the number of memory pages and swap-file disk blocks that are currently unused.

freemem	This shows the average number of memory pages available to user processes over the intervals sampled by the command. Page size is machine-dependent.
freeswap	This shows the number of 512-byte disk blocks available for page swapping.

An example of sar -r output follows:

Solaris mysys Solaris 2.3 sun4c 08/22/93 14:28:12 freemem freeswap 14:29:12 268 3034 14:30:12 351 3009 14:31:12 297 3033 Average 306 3025

Checking CPU Utilization With sar -u

CPU utilization is listed by sar -u. (sar without any options is equivalent to sar -u.) At any given moment, the processor is either busy or idle. When busy, the processor is in either user or system mode. When idle, the processor is either waiting for I/O completion or "sitting still" with no work to do.

The sar -u command displays the following information:

• %sys lists the percentage of time that the processor is in system mode
• %user lists the percentage of time that the processor is in user mode
• %wio lists the percentage of time the processor is idle and waiting for I/O completion
• %idle lists the percentage of time the processor is idle and is not waiting for I/O

A high %wio generally means a disk slowdown has occurred.

The following is an example of sar -u output:

Solaris mysys Solaris 2.3 sun4c 08/22/93 14:28:12 %usr %sys %wio %idle 14:29:12 22 27 18 32 14:30:12 6 24 13 57 14:31:12 8 28 19 45 Average 12 27 17 45

Checking System Table Status With sar -v

The -v option reports the status of the process table, inode table, file table, and shared memory record table.

proc-sz	The number of process entries (proc structs) currently being used, or allocated in the kernel.
inod-sz	The total number of inodes in memory verses the maximum number of inodes allocated in the kernel.

This is not a strict high water mark; it can overflow.

file-sz	The size of the open system file table. The sz is given as 0, since space is allocated dynamically for the file table.
ov	This is the number of times a table has overflowed (reported for the three tables listed above).
lock-sz	The number of shared memory record table entries currently being used or allocated in the kernel. The sz is given as 0 because space is allocated dynamically for the shared memory record table.

An example of sar -v output follows:

Solaris mysys Solaris 2.3 sun4c 08/22/93 14:28:12 proc-sz ov inod-sz ov file-sz ov lock-sz 14:29:12 28/200 0 297/300 0 63/0 0 6/0 14:30:12 30/200 0 297/300 0 65/0 0 6/0 14:31:12 28/200 0 296/300 0 63/0 0 6/0

This example shows that all tables are large enough to have no overflows. These tables are all dynamically allocated based on the amount of physical memory.

Checking Swap Activity With sar -w

The -w option reports swapping and switching activity. The following are some target values and observations:

swpin/s	The number of LWP transfers into memory per second.
bswin/s	The number of 512-byte blocks transferred for swap-ins per second.

---

Note - All process swap-ins include process initialization.

---

swpot/s	The average number of processes swapped out of memory, per second. If the number is greater than 1, you may need to increase memory.
bswot/s	The number of blocks transferred for swap-outs per second.
pswch/s	The number of kernel thread switches per second.

An example of sar -w output follows:

Solaris mysys Solaris 2.3 sun4c 08/22/93 14:28:12 swpin/s pswin/s swpot/s pswot/s pswch/s 14:29:12 0.00 0.0 0.00 0.0 22 14:30:12 0.00 0.0 0.00 0.0 12 14:31:12 0.00 0.0 0.00 0.0 18 Average 0.00 0.0 0.00 0.0 18

Checking Terminal Activity with sar -y

The -y option monitors terminal device activities. If you have a lot of terminal I/O, you can use this report to determine if there are any bad lines. The activities recorded are defined as follows:

rawch/s	Input characters (raw queue), per second.
canch/s	Input characters processed by canon (canonical queue) per second.
outch/s	Output characters (output queue) per second.
rcvin/s	Receiver hardware interrupts per second.
xmtin/s	Transmitter hardware interrupts per second.

mdmin/s             Modem interrupts per second.

The number of modem interrupts per second (mdmin/s) should be close to zero, and the receive and transmit interrupts per second (xmtin/s and rcvin/s) should be less than or equal to the number of incoming or outgoing characters, respectively. If this is not the case, check for bad lines.

An example of sar -y output follows:

Solaris mysys Solaris 2.3 sun4c 08/22/93 14:28:12 rawch/s canch/s outch/s rcvin/s xmtin/s mdmin/s 14:29:12 0 1 157 1 3 0 14:30:12 0 2 34 2 2 0 14:31:12 0 1 11 1 2 0 Average 0 1 67 1 2 0

Checking Overall System Performance With sar -A

The -A option provides a view of overall system performance. Use it to get a more global perspective. If data from more than one time segment is shown, the report includes averages.

Instructions for Monitoring Performance

This section describes how to set up the sadc and sar utilities to monitor the performance of your system.

· How to Set Up Automatic Data Collection

1. As root, open the file /etc/init.d/perf for editing.

2. Verify that the following line is uncommented:

# su sys -c "/usr/lib/sa/sadc /var/adm/sa/sa'date +%d'"

This version of the sadc command writes a special record that marks the time when the counters are reset to zero (boot time). The output of sadc is put into the file sadd (where dd is the current date), which acts as the daily system activity record.

3. Open the file /var/spool/cron/crontabs/sys for editing.

4. Uncomment the following lines:

# 0 * * * 0-6 /usr/lib/sa/sa1
# 20,40 8-17 * * 1-5 /usr/lib/sa/sa1

The first entry writes a record to /var/adm/sa/sadd on the hour, every hour, seven days a week.

The second entry writes a record to /var/adm/sa/sadd twice each hour during peak working hours: at 20 minutes and 40 minutes past the hour, from 8 a.m. to 5 p.m., Monday through Friday.

Thus, these two crontab entries cause a record to be written to /var/adm/sa/sadd every 20 minutes from 8 a.m. to 5 p.m., Monday through Friday, and every hour on the hour otherwise. You can change these defaults to meet your needs.

· How to Display Statistics With vmstat

To display statistics gathered at five-second intervals: * Type vmstat 5 and press Return.

To display statistics since the last boot: * Type vmstat -s and press Return.

To display swapping activity: * Type vmstat -S and press Return.

To display cache flushing statistics: * Type vmstat -c and press Return.

· How to Display I/O Statistics With iostat

To display disk statistics gathered at five-second intervals: * Type iostat 5 and press Return.

Reference Material for Monitoring Performance

Table 8-2 lists the sar command options according to the type of information that they provide:

Table 8-2 sar

Resource	Option	Activity Reported
I/O	-a	File access
	-b	Buffers
	-d	Disk transfers
	-y	Terminal activity
CPU	-c	System calls
	-m	Interprocess communication
	-q	Queue activity
	-u	CPU mode
Memory	-g	Page-out activity
	-k	Memory allocation
	-p	Page-in activity
	-r	Unused memory
	-v	System tables
	-w	Swapping