Chapter 5 Kernel Debugging Modules

This chapter describes the debugger modules, dcmds, and walkers provided to debug the Solaris kernel. Each kernel debugger module is named after the corresponding Solaris kernel module, so that it will be loaded automatically by MDB. The facilities described here reflect the current Solaris kernel implementation and are subject to change in the future; writing shell scripts that depend on the output of these commands is not recommended. In general, the kernel debugging facilities described in this chapter are meaningful only in the context of the corresponding kernel subsystem implementation. See "Related Books and Papers" for a list of references that provide more information about the Solaris kernel implementation.

---

Note -

This guide reflects the Solaris 8 operating environment implementation; these modules, dcmds, and walkers may not be relevant, correct, or applicable to past or future releases, since they reflect the current kernel implementation. They do not define a permanent public interface of any kind. All of the information provided about modules, dcmds, walkers, and their output formats and arguments is subject to change in future releases of the Solaris operating environment.

---

Generic Kernel Debugging Support (genunix)

Kernel Memory Allocator

This section discusses the dcmds and walkers used to debug problems identified by the Solaris kernel memory allocator and to examine memory and memory usage. The dcmds and walkers described here are discussed in more detail in Chapter 6, Debugging With the Kernel Memory Allocator.

dcmds

thread ::allocdby

Given the address of a kernel thread, print a list of memory allocations it has performed in reverse chronological order.

bufctl ::bufctl [-a address] [-c caller] [-e earliest] [-l latest] [-t thread]

Print a summary of the bufctl information for the specified bufctl address. If one or more options are present, the bufctl information is printed only if it matches the criteria defined by the option arguments; in this way, the dcmd can be used as a filter for input from a pipeline. The -a option indicates that the bufctl's corresponding buffer address must equal the specified address. The -c option indicates that a program counter value from the specified caller must be present in the bufctl's saved stack trace. The -e option indicates that the bufctl's timestamp must be greater than or equal to the specified earliest timestamp. The -l option indicates that the bufctl's timestamp must be less than or equal to the specified latest timestamp. The -t option indicates that the bufctl's thread pointer must be equal to the specified thread address.

[ address ] ::findleaks [-v]

The ::findleaks dcmd provides powerful and efficient detection of memory leaks in kernel crash dumps where the full set of kmem debug features has been enabled. The first execution of ::findleaks processes the dump for memory leaks (this can take a few minutes), then coalesces the leaks by the allocation stack trace. The findleaks report shows a bufctl address and the topmost stack frame for each memory leak that was identified.

If the -v option is specified, the dcmd prints more verbose messages as it executes. If an explicit address is specified prior to the dcmd, the report is filtered and only leaks whose allocation stack traces contain the specified function address are displayed.

thread ::freedby

Given the address of a kernel thread, print a list of memory frees it has performed, in reverse chronological order.

value ::kgrep

Search the kernel address space for pointer-aligned addresses that contain the specified pointer-sized value. The list of addresses that contain matching values is then printed. Unlike MDB's built-in search operators, ::kgrep searches every segment of the kernel's address space and searches across discontiguous segment boundaries. On large kernels, ::kgrep can take a considerable amount of time to execute.

::kmalog [ slab | fail ]

Display events in a kernel memory allocator transaction log. Events are displayed in time-reverse order, with the most recent event displayed first. For each event, ::kmalog displays the time relative to the most recent event in T-minus notation (for example, T-0.000151879), the bufctl, the buffer address, the kmem cache name, and the stack trace at the time of the event. Without arguments, ::kmalog displays the kmem transaction log, which is present only if KMF_AUDIT is set in kmem_flags. ::kmalog fail displays the allocation failure log, which is always present; this can be useful in debugging drivers that don't cope with allocation failure correctly. ::kmalog slab displays the slab create log, which is always present. ::kmalog slab can be useful when searching for memory leaks.

::kmastat

Display the list of kernel memory allocator caches and virtual memory arenas, along with corresponding statistics.

::kmausers [-ef] [cache ...]

Print information about the medium and large users of the kernel memory allocator that have current memory allocations. The output consists of one entry for each unique stack trace specifying the total amount of memory and number of allocations that was made with that stack trace. This dcmd requires that the KMF_AUDIT flag is set in kmem_flags.

If one or more cache names (for example, kmem_alloc_256) are specified, the scan of memory usage is restricted to those caches. By default all caches are included. If the -e option is used, the small users of the allocator are included. The small users are allocations that total less than 1024 bytes of memory or for which there are less than 10 allocations with the same stack trace. If the -f option is used, the stack traces are printed for each individual allocation.

[ address ] ::kmem_cache

Format and display the kmem_cache structure stored at the specified address, or the complete set of active kmem_cache structures.

::kmem_log

Display the complete set of kmem transaction logs, sorted in reverse chronological order. This dcmd uses a more concise tabular output format than ::kmalog.

[ address ] ::kmem_verify

Verify the integrity of the kmem_cache structure stored at the specified address, or the complete set of active kmem_cache structures. If an explicit cache address is specified, the dcmd displays more verbose information regarding errors; otherwise, a summary report is displayed. The ::kmem_verify dcmd is discussed in more detail in "Kernel Memory Caches".

[ address ] ::vmem

Format and display the vmem structure stored at the specified address, or the complete set of active vmem structures. This structure is defined in <sys/vmem_impl.h>.

address ::vmem_seg

Format and display the vmem_seg structure stored at the specified address. This structure is defined in <sys/vmem_impl.h>.

address ::whatis [-abv]

Report information about the specified address. In particular, ::whatis will attempt to determine if the address is a pointer to a kmem-managed buffer or another type of special memory region, such as a thread stack, and report its findings. If the -a option is present, the dcmd reports all matches instead of just the first match to its queries. If the -b option is present, the dcmd also attempts to determine if the address is referred to by a known kmem bufctl. If the -v option is present, the dcmd reports its progress as it searches various kernel data structures.

Walkers

allocdby

Given the address of a kthread_t structure as a starting point, iterate over the set of bufctl structures corresponding to memory allocations performed by this kernel thread.

bufctl

Given the address of a kmem_cache_t structure as a starting point, iterate over the set of allocated bufctls associated with this cache.

freectl

Given the address of a kmem_cache_t structure as a starting point, iterate over the set of free bufctls associated with this cache.

freedby

Given the address of a kthread_t structure as a starting point, iterate over the set of bufctl structures corresponding to memory deallocations performed by this kernel thread.

freemem

Given the address of a kmem_cache_t structure as a starting point, iterate over the set of free buffers associated with this cache.

kmem

Given the address of a kmem_cache_t structure as a starting point, iterate over the set of allocated buffers associated with this cache.

kmem_cache

Iterate over the active set of kmem_cache_t structures. This structure is defined in <sys/kmem_impl.h>.

kmem_cpu_cache

Given the address of a kmem_cache_t structure as a starting point, iterate over the per-CPU kmem_cpu_cache_t structures associated with this cache. This structure is defined in <sys/kmem_impl.h>.

kmem_slab

Given the address of a kmem_cache_t structure as a starting point, iterate over the set of associated kmem_slab_t structures. This structure is defined in <sys/kmem_impl.h>.

kmem_log

Iterate over the set of bufctls stored in the kmem allocator transaction log.

File Systems

The MDB file systems debugging support includes a built-in facility to convert vnode pointers to the corresponding file system path name. This conversion is performed using the Directory Name Lookup Cache (DNLC); because the cache does not hold all active vnodes, some vnodes might not be able to be converted to path names and "??" is displayed instead of a name.

dcmds

::fsinfo

Display a table of mounted file systems, including the vfs_t address, ops vector, and mount point of each file system.

::lminfo

Display a table of vnodes with active network locks registered with the lock manager. The pathname corresponding to each vnode is shown.

address ::vnode2path [-v]

Display the pathname corresponding to the given vnode address. If the -v option is specified, the dcmd prints a more verbose display, including the vnode pointer of each intermediate path component.

Walkers

buf

Iterate over the set of active block I/O transfer structures (buf_t structures). The buf structure is defined in <sys/buf.h> and is described in more detail in buf(9S).

Virtual Memory

This section describes the debugging support for the kernel virtual memory subsystem.

dcmds

address ::addr2smap [offset]

Print the smap structure address that corresponds to the given address in the kernel's segmap address space segment.

as ::as2proc

Display the proc_t address for the process corresponding to the as_t address as.

seg ::seg

Format and display the specified address space segment (seg_t address).

vnode ::vnode2smap [offset]

Print the smap structure address that corresponds to the given vnode_t address and offset.

Walkers

anon

Given the address of an anon_map structure as a starting point, iterate over the set of related anon structures. The anon map implementation is defined in <vm/anon.h>.

seg

Given the address of an as_t structure as a starting point, iterate over the set of address space segments (seg structures) associated with the specified address space. The seg structure is defined in <vm/seg.h>.

CPUs and the Dispatcher

This section describes the facilities for examining the state of the cpu structures and the kernel dispatcher.

dcmds

::callout

Display the callout table. The function, argument, and expiration time for each callout is displayed.

::class

Display the scheduling class table.

[ cpuid ] ::cpuinfo [-v]

Display a table of the threads currently executing on each CPU. If an optional CPU ID number is specified prior to the dcmd name, only the information for the specified CPU is displayed. If the -v option is present, ::cpuinfo also displays the runnable threads waiting to execute on each CPU as well as the active interrupt threads.

Walkers

cpu

Iterate over the set of kernel CPU structures. The cpu_t structure is defined in <sys/cpuvar.h>.

Device Drivers and DDI Framework

This section describes dcmds and walkers that are useful for kernel developers as well as third-party device driver developers.

dcmds

::devbindings device-name

Display the list of all instances of the named driver. The output consists of an entry for each instance, beginning with the pointer to the struct dev_info (viewable with $<devinfo or ::devinfo), the driver name, the instance number, and the driver and system properties associated with that instance.

address ::devinfo [ -q ]

Print the system and driver properties associated with a devinfo node. If the -q option is specified, only a quick summary of the device node is shown.

[ address ] ::devnames [ -v ]

Display the kernel's devnames table along with the dn_head pointer, which points at the driver instance list. If the -v flag is specified, additional information stored at each entry in the devnames table is displayed.

[ devinfo ] ::prtconf [ -cpv ]

Display the kernel device tree starting at the device node specified by devinfo. If devinfo is not provided, the root of the device tree is assumed by default. If the -c option is specified, only children of the given device node are displayed. If the -p option is specified, only ancestors of the given device node are displayed. If -v is specified, the properties associated with each node are displayed.

[ major-num ] ::major2name [ major-num ]

Display the driver name corresponding to the specified major number. The major number can be specified as an expression preceding the dcmd or as a command-line argument.

[ address ] ::modctl2devinfo

Print all of the device nodes that correspond to the specified modctl address.

::name2major driver-name

Given a device driver name, display its major number.

[ address ] ::softstate [ instance-number ]

Given a softstate state pointer (see ddi_soft_state_init(9F)) and a device instance number, display the soft state for that instance.

Walkers

devinfo

First, iterate over the parents of the given devinfo and return them in order of seniority from most to least senior. Second, return the given devinfo itself. Third, iterate over the children of the given devinfo in order of seniority from most to least senior. The dev_info struct is defined in <sys/ddi_impldefs.h>.

devinfo_children

First, return the given devinfo, then iterate over the children of the given devinfo in order of seniority from most to least senior. The dev_info struct is defined in <sys/ddi_impldefs.h>.

devinfo_parents

Iterate over the parents of the given devinfo in order of seniority from most to least senior, and then return the given devinfo. The dev_info struct is defined in <sys/ddi_impldefs.h>.

devi_next

Iterate over the siblings of the given devinfo. The dev_info struct is defined in <sys/ddi_impldefs.h>.

devnames

Iterate over the entries in the devnames array. This structure is defined in <sys/autoconf.h>.

STREAMS

This section describes dcmds and walkers that are useful for kernel developers as well as developers of third-party STREAMS modules and drivers.

dcmds

address ::queue [-v] [-f flag] [-F flag] [-m modname]

Filter and display the specified queue_t data structure. With no options, various properties of the queue_t are shown. If the -v option is present, the queue flags are decoded in greater detail. If the -f, -F, or -m options are present, the queue is displayed only if it matches the criteria defined by the arguments to these options; in this way, the dcmd can be used as a filter for input from a pipeline. The -f option indicates that the specified flag (one of the Q flag names from <sys/stream.h>) must be present in the queue flags. The -F option indicates that the specified flag must be absent from the queue flags. The -m option indicates that the module name associated with the queue must match the specified modname.

address ::q2syncq

Given the address of a queue_t, print the address of the corresponding syncq_t data structure.

address ::q2otherq

Given the address of a queue_t, print the address of the peer read or write queue structure.

address ::q2rdq

Given the address of a queue_t, print the address of the corresponding read queue.

address ::q2wrq

Given the address of a queue_t, print the address of the corresponding write queue.

[ address ] ::stream

Display a visual picture of a kernel STREAM data structure, given the address of the stdata_t structure representing the STREAM head. The read and write queue pointers, byte count, and flags for each module are shown, and in some cases additional information for the specific queue is shown in the margin.

address ::syncq [-v] [-f flag] [-F flag] [-t type] [-T type]

Filter and display the specified syncq_t data structure. With no options, various properties of the syncq_t are shown. If the -v option is present, the syncq flags are decoded in greater detail. If the -f, -F, -t, or -T options are present, the syncq is displayed only if it matches the criteria defined by the arguments to these options; in this way, the dcmd can be used as a filter for input from a pipeline. The -f option indicates that the specified flag (one of the SQ_ flag names from <sys/strsubr.h>) must be present in the syncq flags. The -F option indicates that the specified flag must be absent from the syncq flags. The -t option indicates that the specified type (one of the SQ_CI or SQ_CO type names from <sys/strsubr.h>) must be present in the syncq type bits. The -T option indicates that the specified type must be absent from the syncq type bits.

address ::syncq2q

Given the address of a syncq_t, print the address of the corresponding queue_t data structure.

Walkers

qlink

Given the address of a queue_t structure, walk the list of related queues using the q_link pointer. This structure is defined in <sys/stream.h>.

qnext

Given the address of a queue_t structure, walk the list of related queues using the q_next pointer. This structure is defined in <sys/stream.h>.

readq

Given the address of an stdata_t structure, walk the list of read-side queue structures.

writeq

Given the address of an stdata_t structure, walk the list of write-side queue structures.

Files, Processes, and Threads

This section describes dcmds and walkers used to format and examine various fundamental file, process, and thread structures in the Solaris kernel.

dcmds

process ::fd fd-num

Print the file_t address corresponding to the file descriptor fd-num associated with the specified process. The process is specified using the virtual address of its proc_t structure.

thread ::findstack [ command ]

Print the stack trace associated with the given kernel thread, identified by the virtual address of its kthread_t structure. The dcmd employs several different algorithms to locate the appropriate stack backtrace. If an optional command string is specified, the dot variable is reset to the frame pointer address of the topmost stack frame, and the specified command is evaluated as if it had been typed at the command line. The default command string is "<.$C0"; that is, print a stack trace including frame pointers but no arguments.

pid ::pid2proc

Print the proc_t address corresponding to the specified PID. Recall that MDB's default base is hexadecimal, so decimal PIDs obtained using pgrep(1) or ps(1) should be prefixed with 0t.

process ::pmap

Print the memory map of the process indicated by the given process address. The dcmd displays output using a format similar to pmap(1).

[ process ] ::ps [-flt]

Print a summary of the information related to the specified process, or all active system processes, similar to ps(1). If the -f option is specified, the full command name and initial arguments are printed. If the -l option is specified, the LWPs associated with each process are printed. If the -t option is specified, the kernel threads associated with each process LWP are printed.

::ptree

Print a process tree, with child processes indented from their respective parent processes. The dcmd displays output using a format similar to ptree(1).

vnode ::whereopen

Given a vnode_t address, print the proc_t addresses of all processes that have this vnode currently open in their file table.

Walkers

file

Given the address of a proc_t structure as a starting point, iterate over the set of open files (file_t structures) associated with the specified process. The file_t structure is defined in <sys/file.h>.

proc

Iterate over the active process (proc_t) structures. This structure is defined in <sys/proc.h>.

thread

Iterate over a set of kernel thread (kthread_t) structures. If the global walk is invoked, all kernel threads are returned by the walker. If a local walk is invoked using a proc_t address as the starting point, the set of threads associated with the specified process is returned. The kthread_t structure is defined in <sys/thread.h>.

Synchronization Primitives

This section describes dcmds and walkers used to examine particular kernel synchronization primitives. The semantics of each primitive are discussed in the corresponding man pages section 9F: DDI and DKI Kernel Functions.

dcmds

rwlock ::rwlock

Given the address of a readers-writers lock (see rwlock(9F)), display the current state of the lock and the list of waiting threads.

[ address ] ::wchaninfo [-v]

Given the address of a condition variable (see condvar(9F)) or semaphore (see semaphore(9F)), display the current number of waiters on this object. If no explicit address is specified, display all such objects that have waiting threads. If the -v option is specified, display the list of threads that are blocked on each object.

Walkers

blocked

Given the address of a synchronization object (such as a mutex(9F) or rwlock(9F)), iterate over the list of blocked kernel threads.

wchan

Given the address of a condition variable (see condvar(9F)) or semaphore (see semaphore(9F)), iterate over the list of blocked kernel threads.

Cyclics

The cyclic subsystem is a low-level kernel subsystem that provides high resolution, per-CPU interval timer facilities to other kernel services and programming interfaces.

dcmds

::cycinfo [-vV]

Display the cyclic subsystem per-CPU state for each CPU. If the -v option is present, a more verbose display is shown. If the -V option is present, an even more verbose display than -v is shown.

address ::cyclic

Format and display the cyclic_t at the specified address.

::cyccover

Display cyclic subsystem code coverage information. This information is available only in a DEBUG kernel.

::cyctrace

Display cyclic subsystem trace information. This information is available only in a DEBUG kernel.

Walkers

cyccpu

Iterate over the per-CPU cyc_cpu_t structures. This structure is defined in <sys/cyclic_impl.h>.

cyctrace

Iterate over the cyclic trace buffer structures. This information is only available in a DEBUG kernel.

Interprocess Communication Debugging Support (ipc)

The ipc module provides debugging support for the implementation of the message queue, semaphore, and shared memory interprocess communication primitives.

dcmds

::ipcs [-l]

Display a listing of system-wide IPC identifiers, corresponding to known message queues, semaphores, and shared memory segments. If the -l option is specified, a longer listing of information is shown.

[ address ] ::msqid_ds [-l]

Print the specified msqid_ds structure or a table of the active msqid_ds structures (message queue identifiers). If the -l option is specified, a longer listing of information is displayed.

[ address ] ::semid_ds [-l]

Print the specified semid_ds structure or a table of the active semid_ds structures (semaphore identifiers). If the -l option is specified, a longer listing of information is displayed.

[ address ] ::shmid_ds [-l]

Print the specified shmid_ds structure or a table of the active shmid_ds structures (shared memory segment identifiers). If the -l option is specified, a longer listing of information is displayed.

Walkers

msg

Walk the active msqid_ds structures corresponding to message queue identifiers. This structure is defined in <sys/msg.h>.

sem

Walk the active semid_ds structures corresponding to semaphore identifiers. This structure is defined in <sys/sem.h>.

shm

Walk the active shmid_ds structures corresponding to shared memory segment identifiers. This structure is defined in <sys/shm.h>.

Loopback File System Debugging Support (lofs)

The lofs module provides debugging support for the lofs(7FS) file system.

dcmds

[ address ] ::lnode

Print the specified lnode_t, or a table of the active lnode_t structures in the kernel.

address ::lnode2dev

Print the dev_t (vfs_dev) for the underlying loopback mounted filesystem corresponding to the given lnode_t address.

address ::lnode2rdev

Print the dev_t (li_rdev) for the underlying loopback mounted file system corresponding to the given lnode_t address.

Walkers

lnode

Walk the active lnode_t structures in the kernel. This structure is defined in <sys/fs/lofs_node.h>.

Internet Protocol Module Debugging Support (ip)

The ip module provides debugging support for the ip(7P) driver

dcmds

[ address ] ::ire [-q]

Print the specified ire_t, or a table of the active ire_t structures in the kernel. If the -q flag is specified, the send and receive queue pointers are printed instead of the source and destination addresses.

Walkers

ire

Walk the active ire (Internet Route Entry) structures in the kernel. This structure is defined in <inet/ip.h>.

Kernel Runtime Link Editor Debugging Support (krtld)

This section describes the debugging support for the kernel runtime link editor, which is responsible for loading kernel modules and drivers.

dcmds

[ address ] ::modctl

Print the specified modctl, or a table of the active modctl structures in the kernel.

address ::modhdrs

Given the address of a modctl structure, print the module's ELF executable header and section headers.

::modinfo

Print information about the active kernel modules, similar to the output of the /usr/sbin/modinfo command.

Walkers

modctl

Walk the list of active modctl structures in the kernel. This structure is defined in <sys/modctl.h>.

IA: Platform Debugging Support (unix)

These dcmds and walkers are specific to IA.

dcmds

[ cpuid | address ] ::ttrace [-x]

Display trap trace records in reverse chronological order. The trap trace facility is available only in DEBUG kernels. If an explicit dot value is specified, this is interpreted as either a CPU ID number or a trap trace record address, depending on the precise value. If a CPU ID is specified, the output is restricted to the buffer from that CPU. If a record address is specified, only that record is formatted. If the -x option is specified, the complete raw record is displayed.

Walkers

ttrace

Walk the list of trap trace record addresses in reverse chronological order. The trap trace facility is available only in DEBUG kernels.

SPARC: sun4d Platform Debugging Support (unix)

These dcmds and walkers are specific to the SPARC sun4d platform.

dcmds

[ cpuid | address ] ::ttrace [-x]

Display trap trace records in reverse chronological order. The trap trace facility is available only in DEBUG kernels. If an explicit dot value is specified, this is interpreted as either a CPU ID number or a trap trace record address, depending on the precise value. If a CPU ID is specified, the output is restricted to the buffer from that CPU. If a record address is specified, only that record is formatted. If the -x option is specified, the complete raw record is displayed.

Walkers

ttrace

Walk the list of trap trace record addresses in reverse chronological order. The trap trace facility is available only in DEBUG kernels.

SPARC: sun4m Platform Debugging Support (unix)

These dcmds and walkers are specific to the SPARC sun4m platform.

dcmds

[ cpuid | address ] ::ttrace [-x]

Display trap trace records in reverse chronological order. The trap trace facility is available only in DEBUG kernels. If an explicit dot value is specified, this is interpreted as either a CPU ID number or a trap trace record address, depending on the precise value. If a CPU ID is specified, the output is restricted to the buffer from that CPU. If a record address is specified, only that record is formatted. If the -x option is specified, the complete raw record is displayed.

Walkers

ttrace

Walk the list of trap trace record addresses in reverse chronological order. The trap trace facility is only available in DEBUG kernels.

SPARC: sun4u Platform Debugging Support (unix)

These dcmds and walkers are specific to the SPARC sun4u platform.

dcmds

[ address ] ::softint

Display the soft interrupt vector structure at the specified address, or display all the active soft interrupt vectors. The pending count, PIL, argument, and handler function for each structure is displayed.

::ttctl

Display trap trace control records. The trap trace facility is available only in DEBUG kernels.

[ cpuid ] ::ttrace [-x]

Display trap trace records in reverse chronological order. The trap trace facility is available only in DEBUG kernels. If an explicit dot value is specified, this is interpreted as a CPU ID number, and the output is restricted to the buffer from that CPU. If the -x option is specified, the complete raw record is displayed.

[ address ] ::xc_mbox

Display the cross-call mailbox at the specified address, or format all the cross-call mailboxes that have pending requests.

::xctrace

Format and display trace trace records in reverse chronological order that are related to CPU cross-call activity. The trap trace facility is available only in DEBUG kernels.

Walkers

softint

Iterate over the soft interrupt vector table entries.

ttrace

Iterate over the trap trace record addresses in reverse chronological order. The trap trace facility is only available in DEBUG kernels.

xc_mbox

Iterate over the mailboxes used for CPU handshake and cross-call (x-call) requests.