Chapter 23 sysinfo Provider
The sysinfo provider makes available probes that
correspond to kernel statistics classified by the name sys.
Because these statistics provide the input for system monitoring utilities
like mpstat(1M),
the sysinfo provider enables quick exploration of observed
aberrant behavior.
Probes
The sysinfo provider makes available probes that
correspond to the fields in the sys named kernel statistic:
a probe provided by sysinfo fires immediately before the
corresponding sys value is incremented. The following example
shows how to display both the names and the current values of the sys named
kernel statistic using the kstat(1M) command.
$ kstat -n sys
module: cpu instance: 0
name: sys class: misc
bawrite 123
bread 2899
bwrite 17995
...
|
The sysinfo probes are described in Table 23–1.
Table 23–1
sysinfo Probes
|
bawrite
|
Probe that fires whenever a buffer is about to be asynchronously written
out to a device.
|
|
bread
|
Probe that fires whenever a buffer is physically read from a device. bread fires after the buffer has been requested
from the device, but before blocking pending its completion.
|
|
bwrite
|
Probe that fires whenever a buffer is about to be written out to a device,
whether synchronously or asynchronously.
|
|
idlethread
|
Probe that fires whenever a CPU enters the idle loop.
|
|
intrblk
|
Probe that fires whenever an interrupt thread blocks.
|
|
inv_swtch
|
Probe that fires whenever a running thread is forced to involuntarily
give up the CPU.
|
|
lread
|
Probe that fires whenever a buffer is logically read from a device.
|
|
lwrite
|
Probe that fires whenever a buffer is logically written to a device
|
|
modload
|
Probe that fires whenever a kernel module is loaded.
|
|
modunload
|
Probe that fires whenever a kernel module is unloaded.
|
|
msg
|
Probe that fires whenever a msgsnd(2) or msgrcv(2) system call is made, but before
the message queue operations have been performed.
|
|
mutex_adenters
|
Probe that fires whenever an attempt is made to acquire an owned adaptive
lock. If this probe fires, one of the lockstat provider's adaptive-block or adaptive-spin probes will also
fire. See Chapter 18, lockstat Provider for
details.
|
|
namei
|
Probe that fires whenever a name lookup is attempted in the filesystem.
|
|
nthreads
|
Probe that fires whenever a thread is created.
|
|
phread
|
Probe that fires whenever a raw I/O read is about to be performed.
|
|
phwrite
|
Probe that fires whenever a raw I/O write is about to be performed.
|
|
procovf
|
Probe that fires whenever a new process cannot be created because the
system is out of process table entries.
|
|
pswitch
|
Probe that fires whenever a CPU switches from executing one thread to
executing another.
|
|
readch
|
Probe that fires after each successful read, but before control is returned
to the thread performing the read. A read may occur through the read(2), readv(2) or pread(2) system calls. arg0 contains
the number of bytes that were successfully read.
|
|
rw_rdfails
|
Probe that fires whenever an attempt is made to read-lock a readers/writer
when the lock is either held by a writer, or desired by a writer. If this
probe fires, the lockstat provider's rw-block probe
will also fire. See Chapter 18, lockstat Provider for details.
|
|
rw_wrfails
|
Probe that fires whenever an attempt is made to write-lock a readers/writer
lock when the lock is held either by some number of readers or by another
writer. If this probe fires, the lockstat provider's rw-block probe will also fire. See Chapter 18, lockstat Provider for details.
|
|
sema
|
Probe that fires whenever a semop(2) system
call is made, but before any semaphore operations have been performed.
|
|
sysexec
|
Probe that fires whenever an exec(2) system
call is made.
|
|
sysfork
|
Probe that fires whenever a fork(2) system
call is made.
|
|
sysread
|
Probe that fires whenever a read(2), readv(2), or pread(2) system call is made.
|
|
sysvfork
|
Probe that fires whenever a vfork(2) system
call is made.
|
|
syswrite
|
Probe that fires whenever a write(2), writev(2), or pwrite(2) system call is made.
|
|
trap
|
Probe that fires whenever a processor trap occurs. Note that some processors,
in particular UltraSPARC variants, handle some light-weight traps through
a mechanism that does not cause this probe to fire.
|
|
ufsdirblk
|
Probe that fires whenever a directory block is read from the UFS file
system. See ufs(7FS) for
details on UFS.
|
|
ufsiget
|
Probe that fires whenever an inode is retrieved. See ufs(7FS) for details on UFS.
|
|
ufsinopage
|
Probe that fires after an in-core inode without any
associated data pages has been made available for reuse. See ufs(7FS) for details on UFS.
|
|
ufsipage
|
Probe that fires after an in-core inode with associated
data pages has been made available for reuse. This probe fires after the associated
data pages have been flushed to disk. See ufs(7FS) for details on UFS.
|
|
writech
|
Probe that fires after each successful write, but before control is
returned to the thread performing the write. A write may occur through the write(2), writev(2) or pwrite(2) system calls. arg0 contains
the number of bytes that were successfully written.
|
|
xcalls
|
Probe that fires whenever a cross-call is about to be made. A cross-call
is the operating system's mechanism for one CPU to request immediate work
of another CPU.
|
Arguments
The arguments to sysinfo probes are as follows:
|
arg0
|
The value by which the statistic is to be incremented. For most probes,
this argument is always 1, but for some probes this argument may take other
values.
|
|
arg1
|
A pointer to the current value of the statistic to be incremented. This
value is a 64–bit quantity that will be incremented by the value in arg0. Dereferencing this pointer enables consumers to determine
the current count of the statistic corresponding to the probe.
|
|
arg2
|
A pointer to the cpu_t structure that corresponds
to the CPU on which the statistic is to be incremented. This structure is
defined in <sys/cpuvar.h>, but it is part of the kernel
implementation and should be considered Private.
|
The value of arg0 is 1 for most sysinfo probes.
However, the readch and writech probes
set arg0 to the number of bytes read or written, respectively.
This features permits you to determine the size of reads by executable name,
as shown in the following example:
# dtrace -n readch'{@[execname] = quantize(arg0)}'
dtrace: description 'readch' matched 4 probes
^C
xclock
value ------------- Distribution ------------- count
16 | 0
32 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1
64 | 0
acroread
value ------------- Distribution ------------- count
16 | 0
32 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 3
64 | 0
FvwmAuto
value ------------- Distribution ------------- count
2 | 0
4 |@@@@@@@@@@@@@ 13
8 |@@@@@@@@@@@@@@@@@@@@@ 21
16 |@@@@@ 5
32 | 0
xterm
value ------------- Distribution ------------- count
16 | 0
32 |@@@@@@@@@@@@@@@@@@@@@@@@ 19
64 |@@@@@@@@@ 7
128 |@@@@@@ 5
256 | 0
fvwm2
value ------------- Distribution ------------- count
-1 | 0
0 |@@@@@@@@@ 186
1 | 0
2 | 0
4 |@@ 51
8 | 17
16 | 0
32 |@@@@@@@@@@@@@@@@@@@@@@@@@@ 503
64 | 9
128 | 0
Xsun
value ------------- Distribution ------------- count
-1 | 0
0 |@@@@@@@@@@@ 269
1 | 0
2 | 0
4 | 2
8 |@ 31
16 |@@@@@ 128
32 |@@@@@@@ 171
64 |@ 33
128 |@@@ 85
256 |@ 24
512 | 8
1024 | 21
2048 |@ 26
4096 | 21
8192 |@@@@ 94
16384 | 0
|
The sysinfo provider sets arg2 to
be a pointer to a cpu_t, a structure internal to the kernel
implementation. The sysinfo probes fire on the CPU on which
the statistic is being incremented. Use the cpu_id member
of the cpu_t structure to determine the CPU of interest.
Example
Examine the following output from mpstat(1M):
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
12 90 22 5760 422 299 435 26 71 116 11 1372 5 19 17 60
13 46 18 4585 193 162 431 25 69 117 12 1039 3 17 14 66
14 33 13 3186 405 381 397 21 58 105 10 770 2 17 11 70
15 34 19 4769 109 78 417 23 57 115 13 962 3 14 14 69
16 74 16 4421 437 406 448 29 77 111 8 1020 4 23 14 59
17 51 15 4493 139 110 378 23 62 109 9 928 4 18 14 65
18 41 14 4204 494 468 360 23 56 102 9 849 4 17 12 68
19 37 14 4229 115 87 363 22 50 106 10 845 3 15 14 67
20 78 17 5170 200 169 456 26 69 108 9 1119 5 21 25 49
21 53 16 4817 78 51 394 22 56 106 9 978 4 17 22 57
22 32 13 3474 486 463 347 22 48 106 9 769 3 17 17 63
23 43 15 4572 59 34 361 21 46 102 10 947 4 15 22 59
|
From the above output, you might conclude that the xcal field
seems too high, especially given the relative idleness of the system. mpstat determines the value in the xcal field by examining
the xcalls field of the sys kernel statistic.
This aberration can therefore be explored easily by enabling the xcalls sysinfo probe, as shown in the following example:
# dtrace -n xcalls'{@[execname] = count()}'
dtrace: description 'xcalls' matched 4 probes
^C
dtterm 1
nsrd 1
in.mpathd 2
top 3
lockd 4
java_vm 10
ksh 19
iCald.pl6+RPATH 28
nwadmin 30
fsflush 34
nsrindexd 45
in.rlogind 56
in.routed 100
dtrace 153
rpc.rstatd 246
imapd 377
sched 431
nfsd 1227
find 3767
|
The output shows where to look for the source of the cross-calls. Some
number of find(1) processes
are causing the majority of the cross-calls. The following D script can be
used to understand the problem in further detail:
syscall:::entry
/execname == "find"/
{
self->syscall = probefunc;
self->insys = 1;
}
sysinfo:::xcalls
/execname == "find"/
{
@[self->insys ? self->syscall : "<none>"] = count();
}
syscall:::return
/self->insys/
{
self->insys = 0;
self->syscall = NULL;
}
This script uses the syscall provider to attribute
cross-calls from find to a particular system call. Some
cross-calls, such as those resulting from page faults, might not emanate from
system calls. The script prints “<none>”
in these cases. Running the script results in output similar to the following
example:
# dtrace -s ./find.d
dtrace: script './find.d' matched 444 probes
^C
<none> 2
lstat64 2433
getdents64 14873
|
This output indicates that the majority of cross-calls induced by find are in turn induced by getdents(2) system calls. Further exploration
would depend on the direction you want to explore. If you want to understand
why find processes are making calls to getdents,
you could write a D script to aggregate on ustack() when find induces a cross-call. If you want to understand why calls to getdents are inducing cross-calls, you could write a D script to
aggregate on stack() when find induces
a cross-call. Whatever your next step, the presence of the xcalls probe
has enabled you to quickly discover the root cause of the unusual monitoring
output.
Stability
The sysinfo provider uses DTrace's stability mechanism
to describe its stabilities, as shown in the following table. For more information
about the stability mechanism, see Chapter 39, Stability.
|
Element
|
Name stability
|
Data stability
|
Dependency class
|
|
Provider
|
Evolving
|
Evolving
|
ISA
|
|
Module
|
Private
|
Private
|
Unknown
|
|
Function
|
Private
|
Private
|
Unknown
|
|
Name
|
Evolving
|
Evolving
|
ISA
|
|
Arguments
|
Private
|
Private
|
ISA
|
Download this book in PDF (2009 KB)