3.1 Solaris Reconfiguration Reboots

---

Note - It is not necessary to perform a reconfiguration reboot to add disks to an existing SPARCstorage array. Refer to the SPARCcluster High Availibility Server Software Administration Guide for this procedure.

---

Be certain to avoid performing Solaris reconfiguration reboots when any hardware (especially a SPARCstorage array or other disks) is not operational (powered off or otherwise defective). A reconfiguration reboot is performed via the OBP boot -r command or by creating the file /reconfigure on the server and then rebooting. In such situations the reconfiguration reboot will remove the inodes in /devices and symlinks in /dev/dsk and /dev/rdsk associated with the disk devices. These disks will become inaccessible to Solaris until a later reconfiguration reboot. A subsequent reconfiguration reboot may not restore the original controller/minor unit numbering and cause Solstice DiskSuite to reject the disks. Once the original numbering is restored, Solstice Disksuite will be able to access the associated metadevices.

If all hardware is operational a reconfiguration reboot may be safely performed to add a SPARCstorage array to a system. Such arrays must be added symmetrically to both systems (though a temporary unbalance is allowed while the systems are upgraded). Similarly, if all hardware is operational it is safe to perform a reconfiguration reboot to remove hardware.

3.2 SPARCstorage Array/Optical Connections Faults

System console messages indicate a node has a defective disk, controller, or cable. First isolate the fault using the procedures in the following two sections, if the fault matches the section heading. Otherwise proceed to Section 3.2.3, "SPARCstorage Array Fails to Communicate" and proceed as directed.

3.2.1 Multi-disk Errors From Both Nodes on the Same SPARCstorage Array

Figure 3-1

---

Note - The following procedure isolates a probable failure of a SPARCstorage array controller board.

---

1. Check AC and DC lights on disk power supply. Refer to the

   SPARCstorage Array Model 100 Series Service Manual. If light display is normal, proceed to step 2.

2. Contact the system administrator and request that the node be prepared for replacement of a controller in a SPARCstorage array.

3. Bring the SPARCstorage array down as described in Chapter 7, "Shutdown and Restart Procedures."

4. Replace the controller board as described in Chapter 5 of the

   SPARCstorage Array Model 100 Series Service Manual.

5. Bring the SPARCstorage array tray up as described in Chapter 7, "Shutdown and Restart Procedures."

6. Contact the system administrator and indicate that the node is ready to be returned to the cluster following replacement of a controller in a SPARCstorage array.

3.2.2 Multiple Disk Errors or Disk Communication Error For One Node Only

Figure 3-2

---

Note - The following procedure isolates a probable failure of a single disk, or the path to the disk, for example, an optical cable or a fibre channel optical module on the node or on the SPARCstorage array.

---

---

Note - You can remove and replace a disk drive, without powering off the SPARCstorage array. It is necessary only to pull out the tray that the drive is in.

---

1. Contact the system administrator and request that the node be prepared for replacement of a disk.

2. Replace the defective disk drive as described in Chapter 5 of the

   SPARCstorage Array Model 100 Series Service Manual.

3. Contact the system administrator and request that the node be returned to the cluster.

4. If the disk drive errors still exist after replacing the drive, proceed to the next section and isolate the problem to its probable cause.

3.2.3 SPARCstorage Array Fails to Communicate

If a SPARCstorage Array is not communicating with a node, begin troubleshooting by making a physical inspection.

1. Ensure that the SPARCstorage Array subsystem is connected to a working power outlet.

2. Check the power cord connection of the SPARCstorage Array power supply.

3. Check the on/off switch of the power supply.

4. Ensure that the fiber optic cable is properly connected at both the node and SPARCstorage Array subsystem.

If the node and the SPARCstorage Array subsystem are still not communicating, then one of the following components is probably faulty:

• Fiber optic cable connecting the node to the SPARCstorage array
• FC/OM on the node
• Fibre Channel SBus (FC/S) card on the node
• FC/OM on the SPARCstorage array
• Array controller on the SPARCstorage array

Follow these instructions to determine if one of the preceding components has failed:

1. Contact the system administrator and request that the node be prepared for troubleshooting, which will require the shutdown of a SPARCstorage array.

2. Shutdown the SPARCstorage array as described in Chapter 7, "Shutdown and Restart Procedures."

3. Go to the controller board at the rear of the SPARCstorage array and set the DIAG switch to DIAG EXT.

   Setting the DIAG switch to DIAG EXT provides more thorough testing, but it also causes the array to take longer to boot.

4. Press the Reset switch to reset the SPARCstorage array.

5. Check the front panel LCD display and see if a SPARCstorage array-specific POST code is displayed in the alphanumerics portion of the LCD display.

   Figure 3-3 shows the location of the alphanumerics portion of the LCD and Table 3-1 lists the POST codes specific to the SPARCstorage array.

Figure 3-3

Table 3-1

POST Code	Meaning	Action
01	LCD failure	Replace fan tray
08	Fan failure	Replace fan tray
09	Power supply failure	Replace power supply
30	Battery failure	Replace battery module
Any other number	Controller failure	Replace controller

• If you do not see a SPARCstorage array-specific POST code displayed, set the DIAG switch back to DIAG, then go to step 6.
• If you see a SPARCstorage array-specific POST code displayed, set the DIAG switch back to DIAG, then replace the indicated component as described in Chapter 5 of the SPARCstorage Array Model 100 Series Service Manual. Contact the system administrator and indicate that the node is ready to be returned to the cluster following component replacement.

6. Become superuser and shutdown the processor for the node.

   Verify that the system returns to the ok prompt after the shutdown is complete.

   If the system returns to the > prompt after the shutdown, enter n to display the ok prompt.

7. Enter the following commands at the ok prompt:

ok true to diag-switch? ok true to fcode-debug? ok reset

8. The system will immediately boot unless you enter a control ] to get the telnet prompt and then enter the following:

telnet> send break

After the ok prompt displays enter the following:

ok show-devs

The following example is for a SPARCcluster 1000 populated with 2 system boards 0 and 1. System board 0 has a QEC and an FC/S in SBus slots 1 and 3, respectively; system board 1 has a QEC in SBus slot 1 and FC/S(s) in slots 2 and 3. You should see output similar to the following:

/io-unit@f,e1200000/sbi@0,0/SUNW,soc@3,0 /io-unit@f,e1200000/sbi@0,0/SUNW,soc@2,0 /io-unit@f,e1200000/sbi@0,0/qec@1,20000 /io-unit@f,e0200000/sbi@0,0/SUNW,soc@3,0 /io-unit@f,e0200000/sbi@0,0/qec@1,20000

9. Locate the lines in the output that give information on the FC/S cards installed in the node.

   You can find the lines by looking for soc@x,x in the output. The first x in soc@x,x tells you which SBus slot the FC/S card is installed in. For example, looking at the output given above, the first line of the output:

/io-unit@f,e1200000/sbi@0,0/SUNW,soc@3,0

tells you that an FC/S card is installed in SBus slot 3 of system board 1.

10. Locate the FC/S card that is connected to the SPARCstorage array that is not communicating with the node system.

11. Determine what the SBus slot number is for that FC/S card.

    Refer to the processor service manual that came with your system for more information on SBus slot numbers for your system.

• If you can find an entry in the output for the FC/S card installed in that SBus slot, go to Step 12.
• If you cannot find an entry in the output for the FC/S card installed in that SBus slot, replace the FC/S card in that SBus slot according to the instructions given in the processor service manual that came with your system. Following replacement of the FC/S card, contact the system administrator and indicate that the node is ready to be returned to the cluster following component replacement.

12. Enter the following at the ok prompt:

ok path select-dev

where path is the entire path given in the line containing the soc@x,x output. Using the previous output as an example, you would enter:

ok " /io-unit@f,e1200000/sbi@0,0/SUNW,soc@3,0" select-dev

---

Note - From this point on, if you enter a command incorrectly, and you get the error message Level 15 Interrupt or Data Access Exception, then you must enter the command given in step 11 again to select the FC/S card again.

---

13. Enter the following at the ok prompt:

ok soc-post

• If you see a message saying that the test passed, go to step 13.
• If you see a message saying that the test failed, replace the FC/S card in that SBus slot according to the instructions given in the processor service manual that came with your system. Following replacement of the FC/S card, contact the system administrator and indicate that the node is ready to be returned to the cluster following component replacement.

14. Disconnect the fiber optic cable from the FC/OM on the node.

15. Get the loopback connector, Part Number 130-2837-01, from the ship kit and install it in the FC/OM on the node.

16. Enter the following at the ok prompt:

ok 40 is frame-dsize ok 1 is frame-num ok 1 is sb-burst-size

17. Locate the FC/OM(s) in the FC/S card and determine whether the FC/OM(s) are in slot A or B in the FC/S card.

    You should be able to see the letters "A" and "B" silkscreened on the outside of the FC/S card.

18. Probe only off the slots that contain an FC/OM.

    Note - Due to a silkscreening error, the "A" and "B" on the outside of the FC/S card are reversed, so the command to probe off slot A will actually probe off slot B and vice versa.

a. If you have an FC/OM in slot A, enter the following at the ok prompt:

ok soc-txrx-extb

b. If you have an FC/OM in slot B, enter the following at the ok prompt:

ok soc-txrx-exta

• If you see a message saying that the test passed, go to Step 19.
• If you see a message saying that the test failed, then replace the FC/OM from the approriate slot on the FC/S card according to the instructions given in the processor service manual that came with your system. Following replacement of the FC/S card, contact the system administrator and indicate that the node is ready to be returned to the cluster following component replacement.

---

Note - Because the SPARCstorage array diagnostics can only check the FC/OMs on the node, the next steps in this procedure will call for you to switch the FC/OMs from the SPARCstorage array with the FC/OMs from the FC/S card on the node.

---

19. Remove the loopback connector from the FC/OM on the node.

20. Remove the FC/OM(s) from the FC/S card in the node.

    Refer to the processor service manual that came with your system for the necessary instructions.

21. Remove the FC/OM(s) from the SPARCstorage array, taking care to keep them separate from the FC/OM(s) that you just removed from the node.

    Refer to Chapter 5 of the SPARCstorage Array Model 100 Series Service Manual, for those instructions.

22. Install the FC/OM(s) from the SPARCstorage array onto the FC/S card in the node.

23. Install the FC/OM(s) from the FC/S card on the node into the SPARCstorage array.

24. Install the loopback connector on the FC/OM on the node.

25. Probe only off the slots that contain an FC/OM.

    a. If you have an FC/OM in the A slot, enter the following at the ok prompt:

ok soc-txrx-extb

b. If you have an FC/OM installed in the B slot in the FC/S card, enter the following at the ok prompt:

ok soc-txrx-exta

• If you see a message saying that the test passed, go to Step 26.
• If you see a message saying that the test failed, then replace the FC/OM from the approriate slot on the FC/S card according to the instructions given in the processor service manual that came with your system. Following replacement of the FC/OM, contact the system administrator and indicate that the node is ready to be returned to the cluster following component replacement.

26. Replace the fiber optic cable.

    Refer to Chapter 5 of the SPARCstorage Array Model 100 Series Service Manual, for those instructions.

27. Replace the cable and then bring up the applicable SPARCstorage array, refer to Chapter 7, "Shutdown and Restart Procedures."

28. Contact the system administrator and indicate that the node is ready to be returned to the cluster following component replacement.

29. If the node still can't communicate with the SPARCstorage array, contact the system administrator and request that the node be prepared for replacement of a controller in a SPARCstorage array.

30. Bring down the SPARCstorage array, refer to Chapter 7, "Shutdown and Restart Procedures."

31. Replace the array controller.

32. Bring up the applicable SPARCstorage array, refer to Chapter 7, Shutdown and Restart Procedures."

33. Contact the system administrator and indicate that the node is ready to be returned to the cluster following replacement of a controller in a SPARCstorage array.

3.3 Node Failures

3.3.1 System Board, Control Board, and Boot Disk

For a system board, control board, and boot disk failures , messages on the system console will identify the defective node and system board slot. A system board fault can then be further isolated using the prtdiag command as described in Section 3.3.2, "prtdiag Command."

This class of faults can be further isolated by referring directly to the troubleshooting procedures in the respective service manual for the system board. Refer to the SPARCserver 1000 System Service Manual for a SPARCcluster 1000HA system and the SPARCcenter 2000 System Service Manual for a SPARCcluster 2000HA system.

After determining which part is defective, perform the following procedure to replace the part:

1. Contact the system administrator and request that the node be prepared for replacement of a processor part.

2. Once the node has been removed from the cluster, part of the system cabinet may be shut down to replace a defective boot disk, system board, processor module, SBus board, SIMM, and so forth.

   Use the respective system processor shutdown procedures to prevent interrupting other cluster components.

• For the SPARCcluster 1000HA: reference Section 7.1.2, "Processor."
• For the SPARCcluster 2000HA: reference Section 7.2.2, "Processor Shutdown/Startup."

3. Replace the defective device as indicated in the applicable service manual.

   Refer to the SPARCserver 1000 System Service Manual for a SPARCcluster 1000HA system and the SPARCcenter 2000 Service Manual for a SPARCcluster 2000HA system.

4. Bring up the applicable processor, refer to Chapter 7, "Shutdown and Restart Procedures."

5. Contact the system administrator and indicate that the node is ready to be returned to the cluster following replacement of a defective part.

3.3.2 prtdiag Command

The prtdiag command is the preferred tool for locating replaceable board components.

---

Note - prtdiag is a UNIX command. It can only be accessed if the OS is booted and running on the machine in question. prtdiag resides in /usr/kvm.

---

Following is an example of the command and its output, actual output will differ.

$ /usr/kvm/prtdiag System Configuration: Sun Microsystems sun4d SPARCserver 1000E System clock frequency: 50 MHz Memory size: 128Mb Number of XDBuses: 1 CPU Units: Frequency Cache-Size Memory Units: Group Size A: MHz MB B: MHz MB 0: MB 1: MB 2: MB 3: MB --------- --------- ----- ----- ---- - ----- Board0: 60 1.0 60 1.0 128 0 0 0 Board1: 60 1.0 60 1.0 0 0 0 0 ======================SBusCards================================ Board0: SBus clock frequency: 25 MHz 0: dma/esp(scsi) 'SUNW,500-2015' lebuffer/le(network) 'SUNW,500-2015' 1: qec/qe(network) 'SUNW,595-3198' 2: qec/be(network) 'SUNW,270-2450' 3: SUNW,soc '501-2069' Board1: SBus clock frequency: 25 MHz 0: dma/esp(scsi) 'SUNW,500-2015' lebuffer/le(network) 'SUNW,500-2015' 1: SUNW,soc '501-2069' 2: qec/be(network) 'SUNW,270-2450' 3: SUNW,soc '501-2069' No failures found in System ===========================

As shown above, prtdiag displays the status of the system boards and replaceable system board components:

• System boards, by location
• SuperSPARC modules, by number, location, and type (identified as operating speed)
• SIMMs, by quantity and locations (identified by group)
• SBus cards, by location and type

3.3.3 Using probe scsi Command

Use this command to verify operation for a new or replacement SCSI-2 device installed in the system.

1. Contact the system administrator and request that the node be removed from the cluster and subsequently halted.

   Once the system is halted, several system messages are displayed. When the messages finish, one of the following two conditions exists; respond as appropriate:

• If the ok prompt is displayed; proceed to step 3.
• If the screen below is displayed:

2. Type the following:

This yields the ok prompt.

3. Enter the appropriate command to probe the system for SCSI-2 devices.

   a. To probe all SCSI-2 devices installed in the system, type:

After entering the probe-scsi-all command, a list of drives like the one that follows is displayed. Example shown is for a SPARCcluster 1000 with two system boards 0 and 1. System board 0 has a CD player, a tape and two SCSI disks; system board 1 has nothing. The probes are off the onboard SCSI -2 host slot (0).

/io-unit@f,e0200000/sbi@0,0/dma@0,81000/esp@0,80000 Target 0 Unit 0 Disk SEAGATE ST31200N SUN1.05872200568303Copyright (c) 1994 Seagate All rights reserved 0000 Target 1 Unit 0 Disk SEAGATE ST31200N SUN1.05872200471832Copyright (c) 1994 Seagate All rights reserved 0000 Target 5 Unit 0 Removable Tape ARCHIVE Python 28454-XXX4.BL Target 6 Unit 0 Removable Read Only device TOSHIBA XM- 4101TASUNSLCD108404/18/94

b. To confine the probe to SCSI-2 devices hosted by a specific SBus SCSI- 2 host, substitute for variables A and B in the command below, where A is the system board number (0-9 for a SC2000 and 0-3 for a SS1000) and B is the SCSI-2 host slot (0-3 for the corresponding SBus slot).

---

Note - The SPARCcluster 1000 has no SBus slot 0, this position is reserved for the onboard SCSI-2 host for SCSI tray devices.

---

For example:

ok probe-scsi /io-unit@f,eA200000/sbi@0,0/dma@B,81000

4. Verify the drive in question is listed.

   The Target # lines identify the SCSI-2 addresses of installed devices. If the address is listed for the device in question, installation was successful. If the address is absent, verify that the cables are installed correctly. If the cabling is correct, run the appropriate diagnostics as required.

5. Reboot the system using the command:

The screen goes blank for several seconds as the system reboots.

3.4 Network Failures

3.4.1 Private Net Failure (SunFastEthernet)

---

Caution - Problems on the private nets may be due to temporary communication conditions. A fix on the private net must be verified, with before and after traffic condition measurements, to verify that comparable traffic has been supported. Do not close a problem by a cable replacement without running netstat before and after the fix, saving the output to a mail message to the support organization for record. Compare the traffic conditions in the two netstat outputs for similar levels.

---

System console messages indicate that one of the private nets has failed. For example, the output of the hastat command will indicate if there are problems with the private nets. Also, the Message Log (at the bottom of the hastat display output) or the /var/adm/messages file should be checked for private net related error messages. The use of the hastat command and the /var/adm/messages file is detailed in Chapter 3 of the SPARCcluster High Availability Server Software Administration Guide.

Supplemental troubleshooting can be found in the SunFastEthernet Adapter User Guide. Also, see Section 3.4.1.1, "One or Both Nodes Up and Running in a Cluster."

3.4.1.1 One or Both Nodes Up and Running in a Cluster

In the following example (see Figure 3-4) both nodes are up and running in a cluster, private net 0 has failed and the HA software has recovered on private net 1.

To find the actual designations for the private net ports on a node:

* Use the netstat -i command on each node to determine which private links be0 and/or be1 are available. For example, for node 0:

node0# netstat -i
Name Mtu   Net/Dest       Address        Ipkts   Ierrs    0pkts   0errs Collis  Queue
be0  1500 204.152.64.0   node0-priv0     642650   0       266563   1      25477   0
be1  1500 204.152.65.0   node0-priv1     642650   0       266563   1      25477   0
-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------------

Figure 3-4

To troubleshoot private net 0 to a defective card or cable, use the following procedure:

n

Note - In the following procedure, node1 is removed from the cluster. When there is one node remaining in a cluster, HA software will continue sending messages across the private nets. The following procedure uses these message packets to confirm communication between the nodes.

1. Contact the system administrator and request that a node be prepared for removal from the cluster.

2. For this example, assume that the HA software recovers on node 1. Refer to Figure 3-5 and remove the private net 1 cable (cable between the be1 ports of both nodes).

3. Connect the private net 0 cable (cable for failed net) between the be0 port of node 0 and the be1 port of node 1.

Figure 3-5

4. Use the snoop command on node 1 as follows:

node1# snoop -d be1

5. If the following string is returned as a result of the snoop command, then the SBus card for the be0 port on node 1 is most likely defective.This message string indicates that the be0 port of node 0 as well as the private net 0 cable are functional.

   In this instance, request that the system administrator rejoin node0 to the cluster and then remove node1 prior to replacing the related SBus card. Once the card is replaced, indicate to the system administrator that node 1 is ready to be returned to the cluster.

node0-priv0 > node1-priv1 UDP D=6666 S=6666 LEN=120

Figure 3-6

6. If the string indicated in step 5 is not returned by the snoop command, then connect the private net 0 cable between the be1 ports of both nodes. Following this, continue using the snoop command on node 1 (snoop will be run as initiated in step 4 until interrupted by a CTRL C.

7. If the message string indicated in step 5 is repeated, then the be0 port on node 0 is most likely defective, as this message indicates that the private net 0 cable is functional.

   In this instance, replace the related SBus card in node 0 and then indicate to the system administrator that node 0 is ready to be returned to the cluster.

8. If the message string indicated in step 5 is not returned, then the private net 0 cable is most likely defective.

3.4.2 Client Net Failure

Messages on the system console will identify the specific port that has failed. Otherwise, for information on test commands as well as additional troubleshooting, refer to the documentation that came with your client network interface card.

3.5 Terminal Concentrator/Serial Connection Faults

Note - It is not necessary for either node to be stopped or removed from a cluster when replacing the terminal concentrator.

Terminal concentrator faults can be isolated using the diagrams depicted in Section 3.5.4, "Terminal Concentrator Flow Diagrams" as well as the information contained in Section 3.5.5, "Additional Troubleshooting Tips."

Figure 3-7

3.5.1 System Indicators

Figure 3-7 depicts the location of terminal concentrator system, test, and status indicators. The system indicators are:

• Power - ON if unit is receiving AC power and the internal DC power supply is working.
• Unit - ON if unit successfully passes its self-test.
• Net - ON when unit successfully transmits test data to and receives test data from the network.
• Attn - ON when unit requires operator attention. Flashing when unit encounters a problem.
• Load - ON when the unit is loading or dumping. Flashing when unit is trying to initiate a load.
• Active - FLASHING when unit successfully transmits data to and receives data from the network; flashing during diagnostics.

The test indicator is located next to the test switch. The indicator lights when the terminal concentrator enters test mode.

The status indicators, numbered 1 to 8, display serial port activity during normal operations. When the terminal concentrator is first configured during the SPARCcluster installation, the indicators should all be OFF. If any status indicator lights, there may be a hardware failure.

3.5.2 Serial Connections

Serial connections between the terminal concentrator and each node may be isolated by the troubleshooting flow diagrams in Section 3.5.4, "Terminal Concentrator Flow Diagrams."

3.5.3 Intermittent Router Problems

If you experience the following:

• Terminal concentrator connections made via routers exhibit intermittent problems while connections from hosts on the same network as the terminal concentrator continue to work normally.
• Also, the terminal concentrator shows no signs of rebooting.

The solution to this problem is to establish a default route within the terminal concentrator and disable the routed feature. The routed feature must be disabled to prevent the default route from being lost. The procedure is as follows:

1. Telnet to the terminal concentrator and become superuser as follows:

$ telnet ss-tc Trying terminal concentrator.. Connected to ss-tc. Escape character is '^]'. Rotaries Defined: cli Enter Annex port name or number: cli Annex Command Line Interpreter * Copyright 1991 Xylogics, Inc. annex: su Password: annex#

2. At the terminal concentrator prompt enter the following:

   annex# edit config.annex

   You should see the following as the first line of help text on a screen editor:

   Ctrl-W: save and exit Ctrl-X: exit Ctrl-F: page down Ctrl-B: page up

   To establish a default route within the terminal concentrator enter the following, where default_router is the IP address for your router:

   %gateway net default gateway default_router metric 1 hardwire

   Follow this with a carriage return and then Ctrl-W to save and exit.

3. Disable the router feature using the set command as follows:

   annex# admin set annex routed n

4. Boot the terminal concentrator:

   annex# boot

3.5.4 Terminal Concentrator Flow Diagrams

Figure 3-8

Figure 3-9

Figure 3-10

Figure 3-11

Figure 3-12

3.5.5 Additional Troubleshooting Tips

3.5.5.1 Terminal Concentrator Indicators

The eight status indicators on the terminal concentrator (Figure 3-7) indicate activity on the serial ports (after POST has passed). Messages from the node should cause the appropriate port LED (2 through 5) to blink. Text entered into the administration workstation should also cause the LED to blink. This can be useful when trying to determine whether the terminal concentrator, node, or cable is bad.

3.5.5.2 Terminal Concentrator System Information

The ROM monitor command, config, allows you to verify the hardware and software revisions of the terminal concentrator. Press the reset button, and after 5 seconds, press the test button.

When the monitor:: prompt appears, type:

monitor: : config <return> REVISION/CONFIGURATION INFORMATION Amount of memory 2 Meg Board ID 52 - Serial Number 172743 REV ROM: Maj Rev 40 Min Rev 0 ROM Software Rev # 0601 MLB Type: 8s,V24,FMC,(1) EXPANSION Type: None,(15) EEPROM size: 32768 bytes FLASH PROM (1048576 bytes) is installed PARITY option is not installed Twisted Pair alternate interface installed Number of ports 3