Appendix H Converting Device Drivers to Support Hotplugging

This appendix provides the information device driver writers need to convert device drivers to be hotplug-capable, which is applicable to the Solaris 7 operating environment.

Introduction

This appendix offers a basic guide to writing drivers which can function in a hotplugging environment. The focus is on bus interfaces which provide a controlled environment for hot removal and insertion of cards.

The focus is on bus interfaces which provide a controlled environment for hot removal and insertion of cards. Before you can convert a driver to be hotplug-capable, you need to understand hotplug scenarios and how they affect device drivers. "Solaris Hotplugging Driver Issues" describes the requirements on device drivers and the flow of control during hot removal and insertion. "Hotplug-Capable Device Driver Development" outlines the steps required to make a device driver hotplug-capable. "Device Driver Testing" provides guidelines for testing hotplug-capable device drivers in Solaris 7.

For more information refer to the Solaris Sample Drivers. This package contains a number of sample drivers that are hotplug-capable and provide greater insight into the process.

Overview of Hotplugging

This section summarizes the sequence of events in hot removal and hot insertion that are relevant to the device driver.

Hotplugging typically involves:

1. Preparation: Indicate to the system that a hotplug operation will take place on particular hardware.

2. Isolation: Specify that the hardware be isolated in preparation for a safe insertion or removal.

3. Removal, insertion, or both: Insert and/or replace the hardware.

4. Verification: Verify that the card is accessible and perform minimal sanity checks.

5. Integration: Re-integrate the hardware into the system.

6. Configuration: Use kernel functions to configure the system, loading and attaching device drivers for newly installed hardware, and deconfiguring the system for removed hardware

Hot Removal With No Replacement

This operation is used to remove a card. You may want to remove a card because, for example, its failure mode affects system operation. To perform a hotplug removal:

1. Indicate, using an administrative application, that a card needs to be removed or replaced.

2. Through the administrative application, use kernel functions to take the appropriate driver offline.

   If the driver instance is busy or open, this operation may fail and you will first have to terminate the processes that are accessing the device.

3. Through the administrative application, use kernel functions to turn off the appropriate slot; an optional slot-state indicator will show that the slot is off and it is now safe to remove the card.

4. Remove the card.

5. Inform the administrative application that the card has been removed.

Hot Insertion

This procedure is used to insert a card in a previously unused slot.

1. Indicate, using an administrative application, that a card is to be inserted in a particular slot.

2. Through the administrative application, use kernel functions to turn off the appropriate slot (if it isn't off already). An optional slot-state indicator may show that the slot is off.

3. Insert the card.

4. Indicate to the administrative application that the card has been inserted.

5. Through the administrative application, use kernel functions to turn on the slot and the slot-state indicator.

6. Through the kernel, configure the system, and load and attach the device driver for the newly installed hardware.

Hot Removal Followed by Insertion

This procedure is used to replace a card with either an identical or a different card.

1. Indicate, using an administrative application, that a card is to be removed or replaced in a given slot.

2. Through the administrative application, use kernel functions to place the appropriate driver offline.

   If the driver instance is busy or open, this operation may fail and you will first have to terminate any processes that are accessing the device.

3. Use the administrative application to disable the appropriate slot, and turn on an optional slot-state indicator to indicate the physical slot and verify that it is now safe to remove the card.

4. Remove the card.

5. Insert the replacement card.

6. Indicate to the administrative application that a card has been inserted.

7. Through the administrative application, use kernel functions to turn on the slot, and, optionally, turn off the slot-state indicator.

8. Through the kernel, configure the system, and load and attach the device driver for the newly installed hardware.

Solaris Hotplugging Driver Issues

This section describes the flow of control from a device driver perspective. Most device drivers can support hotplugging with minimal changes.

Hot Removal

The kernel checks the driver state, and if the driver instance is not open, the kernel can invoke the driver's detach(9E) entry point with the command DDI_DETACH. The driver instance checks its internal state and verifies whether it is safe to detach. If so, the driver frees resources, cancels timeouts, and returns success. At this point, the kernel can remove power from the slot and the hardware can be removed.

Hot Insertion

Hot insertion uses the normal probe(9E) and attach(9E) entry points of a driver after the hardware has been inserted into the slot in a controlled manner.

Hotplug-Capable Device Driver Development

This section briefly discusses how to make Solaris device drivers compliant with the new Solaris technologies of Dynamic Reconfiguration and future Solaris hotplug technologies.

D_HOTPLUG Flag

Hotplug-capable drivers should set D_HOTPLUG in their cb_ops cb_flag.

Detach Entry Point

The driver's detach entry point can be called with these commands:

• DDI_DETACH: The system attempts to unload the driver module. This may be because memory is low or the hardware will be removed. There is no information available to the driver to distinguish between these two cases.

• DDI_PM_SUSPEND: This command is issued when the device is being suspended after a period of inactivity. Refer to Writing Device Drivers for more details on device power management.

• DDI_SUSPEND: This command is issued when the entire system is suspended before power is (possibly) removed or for Dynamic Reconfiguration. All incoming or queued requests must be blocked until the system is resumed. Refer to the bst or sst sample drivers for example code.

It is strongly recommended to fully implement support for all detach commands.

Note the following uses are possible:

• DDI_SUSPEND may be followed by DDI_RESUME with no power interruption.

• DDI_PM_SUSPEND may be followed by power interruption, and is followed by DDI_PM_RESUME.

• DDI_PM_SUSPEND may be followed by DDI_SUSPEND, DDI_RESUME, and DDI_PM_RESUME.

• DDI_DETACH may be followed by power interruption; any further references to the device will need to be preceded by a DDI_ATTACH.

During Dynamic Reconfiguration, there are two possible detach scenarios.

1. All the devices on the detaching system board receive a DDI_DETACH, and all other devices keep running; or

2. All the devices on the detaching system board receive a DDI_DETACH. All other devices receive a DDI_SUSPEND followed by a DDI_RESUME.

DDI_DETACH Command

The DDI_DETACH operation is the inverse of DDI_ATTACH. The DDI_DETACH handling should only deallocate the data structures for the specified instance. Driver global resources must only be deallocated in _fini(9E). Since instances are assigned in arbitrary order, the driver must be able to handle out-of-sequence presentation of instances. In fact, no assumption should be made about the instance number.

The DDI_DETACH command code should perform the following actions.

1. Check whether DDI_DETACH is safe.

   The driver can assume that the device has been closed before DDI_DETACH is issued. However, there may be outstanding callbacks that cannot be cancelled, or the device may not currently be in a state that permits it to be reliably shut down and restarted later. While timeouts or callbacks are still active, proper locking must be enforced.

   The driver should not block while waiting for callback completion or for the device to become idle.

   Devices that maintain some state after a close operation must be carefully analyzed. When a driver not currently in use is automatically unloaded (for example, because the system memory is low) and later automatically reloaded when the user opens the device, this might cause undesirable operation.

   For example, a tape driver that supports non-rewinding tape access might fail the detach operation when the tape head is not at the beginning of the tape. If the drive is powered down, the head position will be lost.

2. Shut down the device and disable interrupts.

   A device needs enough hardware information/support to be able to shut off and restart interrupts. This may already be coded in the driver as a function of the existing detach routines.

3. Remove any interrupts registered with the system.

4. Cancel any outstanding timeouts and callbacks.

5. Quiesce or remove any driver threads.

6. Deallocate memory resources.

   The driver should be unloadable without memory leaks.

7. Unmap any mapped device registers.

8. Execute ddi_set_driver_private(dip, NULL).

9. Free the softstate structure for this instance.

When there is failure during detach, the driver must decide whether to continue the detach and return success or undo the detach actions completed to that point and return failure. Undoing might be risky and it is usually preferable to continue the detach operation.

DDI_DETACH may be followed by power interruption; any further references to the device will need to be preceeded by a DDI_ATTACH.

Note the following when using timeout() routines:

• Avoid using them if at all possible-too many are generally an indication of a poorly designed driver.

• Be careful that you do not have multiple instances of the same routine running; for example, a second call to un->un_tid = timeout(XX_to, arg, ticks); causes un_tid to be overwritten, removing the ability to untimeout() the first timeout() routine.

• Self-rescheduling timeout() routines are those routines that contain a call to timeout() to reschedule themselves. These need particular care to kill.

The timeout routine should take the form of:

static void
 
XX_timeout(caddr_t arg)
{
    struct xx *un = (struct un *)arg;
    mutex_enter(&un->un_lock);
 
    .....
 
    XX_start_timeout(un);
 
    mutex_exit(&un->un_lock);
}
static void

XX_start_timeout(struct xx *un)

{
ASSERT(MUTEX_HELD(&un->un_lock));

    if ((un->un_tid == 0) && ((un->un_flags & XXSTOP) ==
                0)){un->un_tid = timeout(XX_to, arg, ticks);
 
    }
}
static void

XX_stop_timeout(struct xx *un)

{
int     tid;
mutex_enter(&un->un_lock);

    if ((tid = un->un_tid) != 0) {

    /* do not reschedule timeout */
 
        un->un_flags |= XXSTOP;
 
        /* do not hold across untimeout() */
 
        mutex_exit(&un->un_lock);
        (void) untimeout(tid);
        mutex_enter(&un->un_lock);
 
        un->un_flags &= ~XXSTOP;
        mutex_exit(&un->un_lock);
    } else {
        mutex_exit(&un->un_lock);
    }   
}

When deallocating memory, always verify first that the pointer is valid:

#if NONONONO

kmem_free(un->un_buf, un->un_buf_len);
#else
if (un->un_buf) {

    kmem_free(un->un_buf, un->un_buf_len);

    un->un_buf = NULL;
}   
#endif

DDI_SUSPEND Command

System power management and the Dynamic Reconfiguration framework pass the command DDI_SUSPEND to the detach(9E) driver entry point to request that the driver save the device hardware state. The driver may fail the suspend operation if outstanding operations cannot be completed soon or aborted, or if non-cancellable callbacks are outstanding, in which case the system will abort the suspend operation. Note that the driver instance may already have been power managed using DDI_PM_SUSPEND.

To process the DDI_SUSPEND command, the driver must:

1. Set a suspended flag to block new operations.

2. Wait until outstanding operations have completed, or abort them if they can be restarted.

3. Block further operations from being initiated until the device is resumed (except for dump(9E) requests). Refer to sample code in Writing Device Drivers and the bst sample driver.

4. Cancel pending callouts such as timeout callbacks, and quiesce or destroy other driver threads.

5. Save any volatile hardware state in memory. This state includes the contents of device registers, and can also include downloaded firmware.

DDI_SUSPEND is always followed by DDI_RESUME. There may or may not be power interruption.

Attach Entry Point

The system calls attach(9E) to attach a device instance to the system or to resume operation after power has been suspended. The driver's attach(9E) entry point should handle these commands:

• DDI_ATTACH: Initialize the device instance.

• DDI_PM_RESUME: Restore the hardware state of a device after the device has been suspended.

• DDI_RESUME: Restore the hardware state of a device after the system has been suspended.

DDI_PM_RESUME Command

When power is restored to the device that was suspended with DDI_PM_SUSPEND, that device is resumed using the DDI_PM_RESUME command. The device driver should restore the hardware state, set up timeouts again if necessary, and enable interrupts again.

The DDI_PM_RESUME code should make no assumptions about the state of the hardware, which may or may not have lost power.

DDI_RESUME Command

When power is restored to the system or the system is unquiesced, each device that was suspended will be resumed using the DDI_RESUME command. The device driver should restore the hardware state, set up timeouts again if necessary, and enable interrupts again.

If the device is still suspended by DDI_PM_SUSPEND, the driver has to enter a state where it will call ddi_dev_is_needed(9F) for any new or pending requests, since an attach(9E) with DDI_PM_RESUME is still forthcoming.

The resume code should make no assumptions about the state of the hardware, which may or may not have lost power.

Special Issues With SCSI HBA Drivers

SCSI HBA drivers usually do not have a cb_ops(9S) structure, so to enable hotplugging, a minimal cb_ops(9S) structure must be created and exported in the dev_ops(9S).

 /*
      * autoconfiguration routines.
      */ 
     static struct dev_ops xx_dev_ops = {
             ....
             &xx_cb_ops,             /* devo_cb_ops */
             ....
     };
     static struct cb_ops xx_cb_ops = {
     nodev,                  /* open */
      nodev,                  /* close */
      nodev,                  /* strategy */
      nodev,                  /* print */
      nodev,                  /* dump */
      nodev,                  /* read */
      nodev,                  /* write */
      nodev,                  /* ioctl */
      nodev,                  /* devmap */
      nodev,                  /* mmap */
      nodev,                  /* segmap */
      nochpoll,               /* poll */
      nodev,                  /* cb_prop_op */
      0,                      /* streamtab  */
      D_MP | D_HOTPLUG,        /* Driver compatibility flag */
      CB_REV,                 /* cb_rev */
      nodev,                  /* async I/O read entry point */
      nodev                   /* async I/O write entry point */
     };

The DDI_DETACH and DDI_SUSPEND/RESUME requirements are very similar to those of leaf drivers described above.

Device Driver Testing

This section provides some hints for driver testing.

Unloading

The driver should be able to sustain heavy driver activity, followed by module unload, without errors or memory leaks. This can be loop tested as follows (using C shell):

% while 1
<run driver test>
modunload -i  <module number>
> end

Alternatively:

% while 1

add_drv  driver name
<run driver test>
rem_drv driver name
> end

Suspend/Resume Testing

Start up a test on your driver and simultaneously run:

# uadmin 3 8

You can also test Suspend/Resume by generating I/O to your device, and then pressing the top-right button on the keyboard to suspend the machine. Then resume the machine; there should be no panics, error messages, or data corruption.

If you have access to a server system that supports Dynamic Reconfiguration, you may also want to test your driver on this configuration.

Conclusion

The Solaris 7 driver framework provides the necessary hooks to make drivers hotplug-capable with minimal changes to the driver code. If a driver already supports the detach commands DDI_DETACH and DDI_SUSPEND and the attach command DDI_RESUME, simply adding the D_HOTPLUG flag to the cb_ops cb_flag makes the driver hotplug-capable.