Chapter 4 Maintaining Cluster Interconnect Hardware

This chapter describes the procedures to maintain cluster interconnect hardware. The procedures in this chapter apply to all interconnects that Sun Cluster software supports:

• Ethernet

• InfiniBand

• SPARC: Peripheral Component Interconnect-Scalable Coherent Interface (PCI-SCI)

• SPARC: Sun Fire Link

This chapter contains the following procedures:

• How to Add an Interconnect Component

• How to Replace an Interconnect Component

• How to Remove an Interconnect Component

• How to Upgrade Transport Adapter Firmware

For more information, see the following documentation:

• For conceptual information about cluster interconnects, see Cluster Interconnect in Sun Cluster Concepts Guide for Solaris OS.

• For information about how to administer cluster interconnects, see Administering the Cluster Interconnects in Sun Cluster System Administration Guide for Solaris OS.

Maintaining Interconnect Hardware in a Running Cluster

The following table lists procedures about how to maintain cluster interconnect hardware.

Table 4–1 Task Map: Maintaining Cluster Interconnect Hardware

Task	Instructions
Add an interconnect component. Interconnect components include the following components: Transport adapter Transport cable Transport junction (switch)	How to Add an Interconnect Component
Replace an interconnect component. Interconnect components include the following components: Transport adapter Transport cable Transport junction (switch)	How to Replace an Interconnect Component
Remove an interconnect component. Interconnect components include the following components: Transport adapter Transport cable Transport junction (switch)	How to Remove an Interconnect Component
Upgrade transport adapter firmware	How to Upgrade Transport Adapter Firmware

How to Add an Interconnect Component

This procedure defines interconnect component as any one of the following components:

• Transport adapter

• Transport cable

• Transport junction (switch)

This section contains the procedure about how to add transport adapters to nodes in a running cluster.

Before You Begin

This procedure relies on the following prerequisites and assumptions:

• Your cluster is operational and all nodes are powered on.

• If Virtual Local Area Networks (VLANs) are configured, more than one cluster might be impacted by removing a transport junction. Ensure that all clusters are prepared for the removal of a transport junction. Also, record the configuration information of the transport junction you plan to replace and configure the new transport junction accordingly.

  For more information about how to configure VLANs, see Configuring VLANs as Private Interconnect Networks.

1. Determine if you need to shut down and power off the node that is to be connected to the interconnect component you are adding.

   • If you are adding a transport junction, you do not need to shut down and power off the node. Proceed to Step 2.

   • If you are adding a transport cable, you do not need to shut down and power off the node. Proceed to Step 2.

   • If your node has Dynamic Reconfiguration (DR) enabled and you are replacing a transport adapter, you do not need to shut down and power off the node. Proceed to Step 2.

   • If your node does not have DR enabled and you are adding a transport adapter, shut down and power off the node with the transport adapter you are adding.

     For the full procedure about how to shut down a node, see Chapter 3, Shutting Down and Booting a Cluster, in Sun Cluster System Administration Guide for Solaris OS.

2. Install the interconnect component.

   • If you are using an Ethernet or InfiniBand interconnect, see How to Install Ethernet or InfiniBand Transport Cables and Transport Junctions for cabling diagrams and considerations.

   • If you are using a PCI-SCI interconnect, see SPARC: How to Install PCI-SCI Transport Cables and Transport Junction for cabling diagrams and considerations.

   • For the procedure about how to install transport adapters or set transport adapter DIP switches, see the documentation that shipped with your host adapter and node hardware.

   • If your interconnect uses jumbo frames, review the requirements in SPARC: Requirements When Using Jumbo Frames and see the Sun GigaSwift documentation for instructions.

3. If you shut down the node in Step 1, perform a reconfiguration boot to update the new Solaris device files and links. Otherwise, skip this step.

See Also

• To increase Oracle Real Application Clusters performance, set the max-vc-number parameter. See SPARC: Installing PCI-SCI Cluster Interconnect Hardware for more information.

• To reconfigure Sun Cluster software with the new interconnect component, see Chapter 6, Administering Cluster Interconnects and Public Networks, in Sun Cluster System Administration Guide for Solaris OS.

How to Replace an Interconnect Component

This procedure defines interconnect component as any one of the following components:

• Transport adapter

• Transport cable

• Transport junction (switch)

---

Caution –

You must maintain at least one cluster interconnect between the nodes of a cluster. The cluster does not function without a working cluster interconnect. You can check the status of the interconnect with the command scstat -W. For more details about checking the status of the cluster interconnect, see How to Check the Status of the Cluster Interconnect in Sun Cluster System Administration Guide for Solaris OS.

---

You might perform this procedure in the following scenarios:

• You need to replace a failed transport adapter.

• You need to replace a failed transport cable.

• You need to replace a failed transport junction.

For conceptual information about transport adapters, transport cables, and transport junction, see Cluster Interconnect in Sun Cluster Concepts Guide for Solaris OS.

Before You Begin

This procedure relies on the following prerequisites and assumptions.

• Install a temporary crossover cable before you remove a transport junction. This temporary crossover cable eliminates the single point of failure that exists when you remove a transport junction.

• Your cluster is operational and all nodes are powered on.

• Identify the interconnect component that you want to replace. Remove that interconnect component from the cluster configuration by using the procedure in How to Remove Cluster Transport Cables, Transport Adapters, and Transport Junctions in Sun Cluster System Administration Guide for Solaris OS.

  ---

  Note –

  PCI-SCI interconnect requires that you remove the interconnect component from the cluster configuration because the SCI driver needs to be updated. If the SCI driver is not updated, the nodes cannot use that interconnect to communicate. Although this practice is optional for all other interconnect technology, remove the interconnect component from the cluster configuration to prevent error messages from appearing on your console.

  ---

• If Virtual Local Area Networks (VLANs) are configured, more than one cluster might be impacted by removing a transport junction. Ensure that all clusters are prepared for the removal of a transport junction. Also, record the configuration information of the transport junction you plan to replace and configure the new transport junction accordingly.

  For more information about how to configure VLANs, see Configuring VLANs as Private Interconnect Networks.

1. Determine if you need to shut down and power off the node that is connected to the interconnect component you are replacing.

   • If you are replacing a transport junction, you do not need to shut down and power off the node. Proceed to Step 2.

   • If you are replacing a transport cable, you do not need to shut down and power off the node. Proceed to Step 2.

   • If your node has DR enabled and you are replacing a transport adapter, you do not need to shut down and power off the node. Proceed to Step 2.

   • If your node does not have DR enabled and you are replacing a transport adapter, shut down and power off the node with the transport adapter you are replacing.

     For the full procedure about how to shut down a node, see Chapter 3, Shutting Down and Booting a Cluster, in Sun Cluster System Administration Guide for Solaris OS.

2. Disconnect the failed interconnect component from other cluster devices.

   For the procedure about how to disconnect cables from transport adapters, see the documentation that shipped with your host adapter and node.

3. Connect the new interconnect component to other cluster devices.

   • If you are replacing an Ethernet or InfiniBand interconnect, see How to Install Ethernet or InfiniBand Transport Cables and Transport Junctions for cabling diagrams and considerations.

   • If you are replacing a PCI-SCI interconnect, see SPARC: How to Install PCI-SCI Transport Cables and Transport Junction for cabling diagrams and considerations.

   • If your interconnect uses jumbo frames, review the requirements in SPARC: Requirements When Using Jumbo Frames and see the Sun GigaSwift documentation for instructions. Refer to ce Sun Ethernet Driver Considerations for details of how to edit the ce.conf file according to the GigaSwift documentation's instructions.

4. If you shut down the node in Step 1, perform a reconfiguration boot to update the new Solaris device files and links. Otherwise, skip this step.

See Also

To reconfigure Sun Cluster software with the new interconnect component, see How to Add Cluster Transport Cables, Transport Adapters, or Transport Junctions in Sun Cluster System Administration Guide for Solaris OS.

How to Remove an Interconnect Component

This procedure defines interconnect component as any one of the following components:

• Transport adapter

• Transport cable

• Transport junction (switch)

---

Caution –

You must maintain at least one cluster interconnect between the nodes of a cluster. The cluster does not function without a working cluster interconnect. You can check the status of the interconnect with the command scstat -W. For more details about checking the status of the cluster interconnect, see How to Check the Status of the Cluster Interconnect in Sun Cluster System Administration Guide for Solaris OS.

---

You might perform this procedure in the following scenarios:

• You need to remove an unused transport adapter.

• You need to remove an unused transport cable.

• You need to remove an unused transport junction.

• You want to migrate from a two–node cluster that uses switches to a point-to-point configuration.

For conceptual information about transport adapters, transport cables, and transport junctions, see Cluster Interconnect in Sun Cluster Concepts Guide for Solaris OS.

Before You Begin

This procedure relies on the following prerequisites and assumptions:

• If you are migrating from a two–node cluster that uses switches to a point-to-point configuration, install a crossover cable before you remove a switch.

• Your cluster is operational and all nodes are powered on.

• Identify the interconnect component that you want to remove. Remove that interconnect component from the cluster configuration by using the procedure in How to Remove Cluster Transport Cables, Transport Adapters, and Transport Junctions in Sun Cluster System Administration Guide for Solaris OS.

  ---

  Note –

  PCI-SCI interconnect requires that you remove the interconnect component from the cluster configuration because the SCI driver needs to be updated. If the SCI driver is not updated, the nodes cannot use that interconnect to communicate. Although this practice is optional for all other interconnect technology, remove the interconnect component from the cluster configuration to prevent error messages from appearing on your console.

  ---

• If you plan to use virtual local area networks (VLANs) in your cluster interconnect, configure the transport junction. For more information about how configure VLANs, see Configuring VLANs as Private Interconnect Networks.

1. Determine if you need to shut down and power off the node that is connected to the interconnect component you are removing.

   • If you are removing a transport junction you, do not need to shut down and power off the node. Proceed to Step 2.

   • If you are removing a transport cable you, do not need to shut down and power off the node. Proceed to Step 2.

   • If your node has DR enabled and you are removing a transport adapter, you do not need to shut down and power off the node. Proceed to Step 2.

   • If your node does not have DR enabled and you are removing a transport adapter, shut down and power off the node with the transport adapter you are removing.

     For the full procedure about how to shut down a node, see Chapter 3, Shutting Down and Booting a Cluster, in Sun Cluster System Administration Guide for Solaris OS.

2. Disconnect the interconnect component from other cluster devices.

   For the procedure about how to disconnect cables from transport adapters, see the documentation that shipped with your host adapter and node.

3. Remove the interconnect component.

   For the procedure about how to remove interconnect component, see the documentation that shipped with your host adapter, nodes, or switch.

4. If you shut down the node in Step 1, perform a reconfiguration boot to update the new Solaris device files and links. Otherwise, skip this step.

See Also

To reconfigure Sun Cluster software with the new interconnect component, see How to Add Cluster Transport Cables, Transport Adapters, or Transport Junctions in Sun Cluster System Administration Guide for Solaris OS.

How to Upgrade Transport Adapter Firmware

You might perform this procedure in the following scenarios:

• You want to use firmware bug fixes.

• You want to use new firmware features.

Use this procedure to update transport adapter firmware.

1. Determine the resource groups and the device groups that are running on the node. This node is the node on which you are upgrading transport adapter firmware.

   # scstat

2. Migrate the resource groups and device groups off the node on which you plan to upgrade the firmware.

   # scswitch -S -h from-node

3. Perform the firmware upgrade.

   For the procedure about how to upgrade your transport adapter firmware, see the patch documentation.

4. Return the resource groups and device groups to their original node.

   # scswitch -z -g resource-group -h nodename # scswitch -z -D device-group -h nodename