Chapter 6 Execution Control

MDB provides facilities for controlling and tracing the execution of live running programs, including both user applications and the live operating system kernel and device drivers. You can use the mdb command to control user processes that are already running, or create new processes under the control of the debugger. You can boot or load kmdb to control the execution of the operating system kernel itself, or debug a device driver. This chapter describes the built-in dcmds that can be used to control target execution. These commands can be used in either mdb or kmdb, except as noted in the descriptions. Additional topics relating only to execution control in kmdb are discussed in Chapter 7, Kernel Execution Control.

Execution Control

MDB provides a simple model of execution control: a target process can be started from within the debugger using ::run, or MDB can attach to an existing process using :A, ::attach, or the -p command-line option (see Chapter 5, Built-in Commands). Alternately, the kernel can be booted using kmdb or kmdb can be loaded afterward. In either case, a list of traced software events can be specified by the user. Each time a traced event occurs in the target program, all threads in the target stop, the thread that triggered the event is chosen as the representative thread, and control returns to the debugger. Once the target program is set running, control can be asynchronously returned to the debugger by typing the user-defined interrupt character (typically Control-C).

A software event is a state transition in the target program that is observed by the debugger. For example, the debugger may observe the transition of a program counter register to a value of interest (a breakpoint) or the delivery of a particular signal.

A software event specifier is a description of a class of software events that is used by the debugger to instrument the target program in order to observe these events. The ::events dcmd is used to list the software event specifiers. A set of standard properties is associated with each event specifier, as described under ::events in Built-in dcmds.

The debugger can observe a variety of different software events, including breakpoints, watchpoints, signals, machine faults, and system calls. New specifiers can be created using ::bp, ::fltbp, :: sigbp, ::sysbp, or ::wp. Each specifier has an associated callback (an MDB command string to execute as if it had been typed at the command prompt) and a set of properties, as described under ::events in Built-in dcmds. Any number of specifiers for the same event may be created, each with different callbacks and properties. The current list of traced events and the properties of the corresponding event specifiers can be displayed using the ::events dcmd. The event specifier properties are defined as part of the description of the ::events and ::evset dcmds, in Built-in dcmds.

The execution control built-in dcmds, described in Built-in dcmds, are always available, but will issue an error message indicating they are not supported if applied to a target that does not support execution control.

Event Callbacks

The ::evset dcmd and event tracing dcmds allow you to associate an event callback (using the -c option) with each event specifier. The event callbacks are strings that represent MDB commands to execute when the corresponding event occurs in the target. These commands are executed as if they had been typed at the command prompt. Prior to executing each callback, the dot variable is set to the value of the representative thread's program counter and the hits variable is set to the number of times this specifier has been matched, including the current match.

If the event callbacks themselves contain one or more commands to continue the target (for example, ::cont or ::step), these commands do not immediately continue the target and wait for it to stop again. Instead, inside of an event callback, the continue dcmds note that a continue operation is now pending, and then return immediately. Therefore, if multiple dcmds are included in an event callback, the step or continue dcmd should be the last command specified. Following the execution of all event callbacks, the target will immediately resume execution if all matching event callbacks requested a continue. If conflicting continue operations are requested, the operation with the highest precedence determines what type of continue will occur. The order of precedence from highest to lowest is: step, step-over (next), step-out, continue.

Thread Support

MDB provides facilities to examine the stacks and registers of each thread associated with the target. The persistent "thread" variable contains the current representative thread identifier. The format of the thread identifier depends on the target. The ::regs and ::fpregs dcmds can be used to examine the register set of the representative thread, or of another thread if its register set is currently available. In addition, the register set of the representative thread is exported as a set of named variables. The user can modify the value of one or more registers by applying the > dcmd to the corresponding named variable.

The MDB kernel target exports the virtual address of the corresponding internal thread structure as the identifier for a given thread. This address corresponds to the kthread_t data structure in the operating system source code. When using kmdb, the CPU identifier for the CPU running kmdb is stored in the cpuid variable.

The MDB process target provides proper support for examination of multi-threaded user processes that use the native lwp_* interfaces, /usr/lib/libthread.so, or /usr/lib/libpthread.so. When debugging a live user process, MDB will detect if a single threaded process dlopens or closes libthread and will automatically adjust its view of the threading model on-the-fly. The process target thread identifiers will correspond to either the lwpid_t, thread_t, or pthread_t of the representative, depending on the threading model used by the application.

If MDB is debugging a user process target and the target makes use of compiler-supported thread-local storage, MDB will automatically evaluate symbol names referring to thread-local storage to the address of the storage corresponding to the current representative thread. The ::tls built-in dcmd can be used to display the value of the symbol for threads other than the representative thread.

Built-in dcmds

[ addr ] ::bp [+/-dDestT] [-c cmd] [-n count] sym ...
addr :b [cmd ... ]

Set a breakpoint at the specified locations. The ::bp dcmd sets a breakpoint at each address or symbol specified, including an optional address specified by an explicit expression preceding the dcmd, and each string or immediate value following the dcmd. The arguments may either be symbol names or immediate values denoting a particular virtual address of interest. If a symbol name is specified, it may refer to a symbol that cannot yet be evaluated in the target process: that is, it may consist of an object name and function name in a load object that has not yet been opened. In this case, the breakpoint is deferred and it will not be active in the target until an object matching the given name is loaded. The breakpoint will be automatically enabled when the load object is opened. Breakpoints on symbols defined in a shared library should always be set using a symbol name and not using an address expression, as the address may refer to the corresponding Procedure Linkage Table (PLT) entry instead of the actual symbol definition. Breakpoints set on PLT entries may be overwritten by the run-time link-editor when the PLT entry is subsequently resolved to the actual symbol definition. The -d, -D, -e, -s, -t, -T, -c, and -n options have the same meaning as they do for the ::evset dcmd, as described later in this section. If the :b form of the dcmd is used, a breakpoint is only set at the virtual address specified by the expression preceding the dcmd. The arguments following the :b dcmd are concatenated together to form the callback string. If this string contains meta-characters, it must be quoted.

function ::call [ arg ... ]

When using kmdb only, call the specified function defined in the operating system kernel. The function expression must match the address of a defined function in a symbol table of one of the known kernel modules. If expression arguments are specified, these arguments as passed by value. If string arguments are specified, these arguments are passed by reference.

---

Note –

The ::call command should be used only with extreme caution and should never be applied to a production system. The operating system kernel will not resume execution in order to execute the specified function. Therefore, the function being called must not utilize arbitrary kernel services and must not block for any reason. You must be fully aware of the side-effects of any function you call using this command.

---

::cont [SIG]
:c [SIG]

Suspend the debugger, continue the target program, and wait for it to terminate or stop following a software event of interest. If the target is already running because the debugger was attached to a running program with the -o nostop option enabled, this dcmd simply waits for the target to terminate or stop after an event of interest. If an optional signal name or number (see the signal (3HEAD) man page) is specified as an argument, the signal is immediately delivered to the target as part of resuming its execution. If the SIGINT signal is traced, control may be asynchronously returned to the debugger by typing the user-defined interrupt character (usually ^C). This SIGINT signal will be automatically cleared and will not be observed by the target the next time it is continued. If no target program is currently running, ::cont will start a new program running as if by ::run.

addr ::delete [id | all]
addr :d [id | all]

Delete the event specifiers with the given id number. The id number argument is interpreted in decimal by default. If an optional address is specified preceding the dcmd, all event specifiers that are associated with the given virtual address are deleted (e.g. all breakpoints or watchpoints affecting that address). If the special argument "all" is given, all event specifiers are deleted, except those that are marked sticky (T flag). The ::events dcmd displays the current list of event specifiers.

::events [-av]
$b [-av]

Display the list of software event specifiers. Each event specifier is assigned a unique ID number that can be used to delete or modify it at a later time. The debugger may also have its own internal events enabled for tracing; these will only be displayed if the -a option is present. If the -v option is present, a more verbose display including the reason for any specifier inactivity will be shown. The following ::events dcmd shows example output:

> ::events
   ID S TA HT LM Description                              Action
----- - -- -- -- ---------------------------------------- -------------
[ 1 ] - T   1  0 stop on SIGINT                           -      
[ 2 ] - T   0  0 stop on SIGQUIT                          -
[ 3 ] - T   0  0 stop on SIGILL                           -
 ...
[ 11] - T   0  0 stop on SIGXCPU                          -
[ 12] - T   0  0 stop on SIGXFSZ                          -
[ 13] -     2  0 stop at libc`printf                      ::echo printf
>

The following discussion explains the meaning of each column. A summary of this information is available using ::help events.

ID

The event specifier identifier. The identifier will be shown in square brackets [ ] if the specifier is enabled, in parentheses ( ) if the specifier is disabled, or in angle brackets < > if the target program is currently stopped on an event that matches the given specifier.

S

The event specifier state. The state will be one of the following symbols:

-	The event specifier is idle. When no target program is running, all specifiers are idle. When the target program is running, a specifier may be idle if it cannot be evaluated (such as a deferred breakpoint in a shared object that is not yet loaded).
+	The event specifier is active. When the target is continued, events of this type will be detected by the debugger.
*	The event specifier is armed. This state means that the target is currently running with instrumentation for this type of event. This state is only visible if the debugger is attached to a running program with the-o nostop option.
!	The event specifier was not armed due to an operating system error. The ::events -v option can be used to display more information about the reason the instrumentation failed.

TA

The Temporary, Sticky, and Automatic event specifier properties. One or more of the following symbols may be shown:

t	The event specifier is temporary, and will be deleted the next time the target stops, regardless of whether it is matched.
T	The event specifier is sticky, and will be not be deleted by ::delete all or :z. The specifier can be deleted by explicitly specifying its id number to::delete.
d	The event specifier will be automatically disabled when the hit count is equal to the hit limit.
D	The event specifier will be automatically deleted when the hit count is equal to the hit limit.
s	The target will automatically stop when the hit count is equal to the hit limit.

HT

The current hit count. This column displays the number of times the corresponding software event has occurred in the target since the creation of this event specifier.

LM

The current hit limit. This column displays the limit on the hit count at which the auto-disable, auto-delete, or auto-stop behavior will take effect. These behaviors can be configured using the ::evset dcmd.

Description

A description of the type of software event that is matched by the given specifier.

Action

The callback string to execute when the corresponding software event occurs. This callback is executed as if it had been typed at the command prompt.

id ::evset [+/-dDestT] [-c cmd] [-n count] id ...

Modify the properties of one or more software event specifiers. The properties are set for each specifier identified by the optional expression preceding the dcmd and an optional list of arguments following the dcmd. The argument list is interpreted as a list of decimal integers, unless an explicit radix is specified. The ::evset dcmd recognizes the following options:

-d

Disable the event specifier when the hit count reaches the hit limit. If the +d form of the option is given, this behavior is disabled. Once an event specifier is disabled, the debugger will remove any corresponding instrumentation and will ignore the corresponding software events until the specifier is subsequently re-enabled. If the -n option is not present, the specifier is disabled immediately.

-D

Delete the event specifier when the hit count reaches the hit limit. If the +D form of the option is given, this behavior is disabled. The -D option takes precedence over the -d option. The hit limit can be configured using the -n option.

-e

Enable the event specifier. If the +e form of the option is given, the specifier is disabled.

-s

Stop the target program when the hit count reaches the hit limit. If the +s form of the option is given, this behavior is disabled. The -s behavior tells the debugger to act as if ::cont were issued following each execution of the specifier's callback, except for the Nth execution, where N is the current value of the specifier's hit limit. The -s option takes precedence over both the -D option and the -d option.

-t

Mark the event specifier as temporary. Temporary specifiers are automatically deleted the next time the target stops, regardless of whether it stopped as the result of a software event corresponding to the given specifier. If the +t form of the option is given, the temporary marker is removed. The -t option takes precedence over the -T option.

-T

Mark the event specifier as sticky. Sticky specifiers will not be deleted by ::delete all or :z. They can be deleted by specifying the corresponding specifier ID as an explicit argument to ::delete. If the +T form of the option is given, the sticky property is removed. The default set of event specifiers are all initially marked sticky.

-c

Execute the specified cmd string each time the corresponding software event occurs in the target program. The current callback string can be displayed using ::events.

-n

Set the current value of the hit limit to count. If no hit limit is currently set and the -n option does not accompany -s or -D, the hit limit will be set to one.

A summary of this information is available using ::help evset.

flt ::fltbp [+/-dDestT] [-c cmd] [-n count] flt ...

Trace the specified machine faults. The faults are identified using an optional fault number preceding the dcmd, or a list of fault names or numbers (see <sys/fault.h>) following the dcmd. The -d, -D, -e, -s, -t, -T, -c, and -n options have the same meaning as they do for the ::evset dcmd. The ::fltbp command applies to user process debugging only.

signal :i

If the target is a live user process, ignore the specified signal and allow it to be delivered transparently to the target. All event specifiers that are tracing delivery of the specified signal will be deleted from the list of traced events. By default, the set of ignored signals is initialized to the complement of the set of signals that cause a process to dump core by default (see the signal(3HEAD) man page), except for SIGINT, which is traced by default. The :i command applies to user process debugging only.

$i

Display the list of signals that are ignored by the debugger and will be handled directly by the target. More information on traced signals can be obtained using the ::events dcmd. The $i command applies to user process debugging only.

::kill
:k

Forcibly terminate the target if it is a live user process. The target will also be forcibly terminated when the debugger exits if it was created by the debugger using ::run. The ::kill command applies to user process debugging only.

$l

Print the LWPID of the representative thread, if the target is a user process.

$L

Print the LWPIDs of each LWP in the target, if the target is a user process.

::next [SIG]
:e [SIG]

Step the target program one instruction, but step over subroutine calls. If an optional signal name or number (see signal(3HEAD) man page) is specified as an argument, the signal is immediately delivered to the target as part of resuming its execution. If no target program is currently running, ::next will start a new program running as if by ::run and stop at the first instruction.

::run [args ... ]
:r [args ... ]

Start a new target program running with the specified arguments and attach to it. The arguments are not interpreted by the shell. If the debugger is already examining a live running program, it will first detach from this program as if by ::release.

[signal] ::sigbp [+/-dDestT] [-c cmd] [-n count] SIG ...
[signal] :t [+/-dDestT] [-c cmd] [-n count] SIG ...

Trace delivery of the specified signals. The signals are identified using an optional signal number preceding the dcmd, or a list of signal names or numbers (see signal(3HEAD)) following the dcmd. The -d, -D, -e, -s, -t, -T, -c, and -n options have the same meaning as they do for the ::evset dcmd. Initially, the set of signals that cause the process to dump core by default (see signal(3HEAD)) and SIGINT are traced. The ::sigbp command applies to user process debugging only.

::step [branch | over | out] [SIG]
:s SIG
:u SIG

Step the target program one instruction. If an optional signal name or number (see the signal(3HEAD) man page) is specified as an argument and the target is a user process, the signal is immediately delivered to the target as part of resuming its execution. If the optional branch argument is specified, the target program will continue until the next instruction that branches the control flow of the processor. The ::step branch feature is only available when using kmdb on x86 systems with appropriate processor-specific features enabled. If the optional over argument is specified, ::step will step over subroutine calls. The ::step over argument is the same as the ::next dcmd. If the optional out argument is specified, the target program will continue until the representative thread returns from the current function. If no target program is currently running, ::step over will start a new program running as if by ::run and stop at the first instruction. The :s dcmd is the same as ::step. The :u dcmd is the same as ::step out.

[syscall] ::sysbp [+/-dDestT] [-io] [-c cmd] [-n count] syscall ...

Trace entry to or exit from the specified system calls. The system calls are identified using an optional system call number preceding the dcmd, or a list of system call names or numbers (see <sys/syscall.h>) following the dcmd. If the -i option is specified (the default), the event specifiers trigger on entry into the kernel for each system call. If the -o option is specified, the event specifiers trigger on exit out from the kernel. The -d, -D, -e, -s, -t, -T, -c, and -n options have the same meaning as they do for the ::evset dcmd. The ::sysbp command applies to user process debugging only.

addr [,len]::wp [+/-dDestT] [-rwx] [-ip] [-c cmd] [-n count]
addr [,len]:a [cmd... ]
addr [,len]:p [cmd... ]
addr [,len]:w [cmd... ]

Set a watchpoint at the specified address. The length in bytes of the watched region may be set by specifying an optional repeat count preceding the dcmd. If no length is explicitly set, the default is one byte. The ::wp dcmd allows the watchpoint to be configured to trigger on any combination of read (-r option), write (-w option), or execute (-x option) access. The -d, -D, -e, -s, -t, -T, -c, and -n options have the same meaning as they do for the ::evset dcmd. When using kmdb on x86 systems only, the -i option can be used to indicate that a watchpoint should be set on the address of an I/O port. When using kmdb only, the -p option can be used to indicate that the specified address should be interpreted as a physical address. The :a dcmd sets a read access watchpoint at the specified address. The :p dcmd sets an execute access watchpoint at the specified address. The :w dcmd sets a write access watchpoint at the specified address. The arguments following the :a. :p, and :w dcmds are concatenated together to form the callback string. If this string contains meta-characters, it must be quoted.

:z

Delete all event specifiers from the list of traced software events. Event specifiers can also be deleted using ::delete.

Interaction With exec

When a controlled user process performs a successful exec(2), the behavior of the debugger is controlled by the ::set -o follow_exec_mode option, as described in Summary of Command-line Options. If the debugger and victim process have the same data model, then the "stop" and "follow" modes determine whether MDB automatically continues the target or returns to the debugger prompt following the exec. If the debugger and victim process have a different data model, then the "follow" behavior causes MDB to automatically re-exec the MDB binary with the appropriate data model and re-attach to the process, still stopped on return from the exec. Not all debugger state is preserved across this re-exec.

If a 32-bit victim process execs a 64-bit program, then "stop" will return to the command prompt, but the debugger will no longer be able to examine the process because it is now using the 64-bit data model. To resume debugging, execute the ::release -a dcmd, quit MDB, and then execute mdb -p pid to re-attach the 64-bit debugger to the process.

If a 64-bit victim process execs a 32-bit program, then "stop" will return to the command prompt, but the debugger will only provide limited capabilities for examining the new process. All built-in dcmds will work as advertised, but loadable dcmds will not since they do not perform data model conversion of structures. The user should release and reattach the debugger to the process as described above in order to restore full debugging capabilities.

Interaction with Job Control

If the debugger is attached to a user process that is stopped by job control (that is, it stopped in response to SIGTSTP, SIGTTIN, or SIGTTOU), the process may not be able to be set running again when it is continued by a continue dcmd. If the victim process is a member of the same session (that is, it shares the same controlling terminal as MDB), MDB will attempt to bring the associated process group to the foreground and continue the process with SIGCONT to resume it from job control stop. When MDB is detached from such a process, it will restore the process group to the background before exiting. If the victim process is not a member of the same session, MDB cannot safely bring the process group to the foreground, so it will continue the process with respect to the debugger but the process will remain stopped by job control. MDB will print a warning in this case, and the user must issue a fg command from the appropriate shell in order to resume the process.

Process Attach and Release

When MDB attaches to a running user process, the process is stopped and remains stopped until one of the continue dcmds is applied, or the debugger quits. If the -o nostop option is enabled prior to attaching the debugger to a process with -p or prior to issuing an ::attach or :A command, MDB will attach to the process but not stop it. While the process is still running, it may be inspected as usual (albeit with inconsistent results) and breakpoints or other tracing flags may be enabled. If the :c or ::cont dcmds are executed while the process is running, the debugger will wait for the process to stop. If no traced software events occur, the user can send an interrupt (^C) after :c or ::cont to force the process to stop and return control to the debugger.

MDB releases the current running process (if any) when the :R, ::release, :r, ::run, $q, or ::quit dcmds are executed, or when the debugger terminates as the result of an EOF or signal. If the process was originally created by the debugger using :r or ::run, it will be forcibly terminated as if by SIGKILL when it is released. If the process was already running prior to attaching MDB to it, it will be set running again when it is released. A process may be released and left stopped and abandoned using the ::release -a option.