Chapter 3 Case Study: Deploying in a High-Availability Environment Over a WAN Using SSL

The case study provided in this chapter explains how the company “Global Telco” implemented Sun Java System Identity Synchronization for Windows in a four-way multimaster replication (MMR) environment over a Wide Area Network (WAN) using the Secure Socket Layer (SSL).

This chapter covers the following topics:

• Global Telco Deployment Information

• Installation and Configuration Overview

• Configuration Walkthrough

• Failover Installation

• Setting Up SSL

• Increasing Connector Worker Threads

• Aligning Primary and Failover Configurations

• Initial idsync resync Operations

• Periodic idsync resync Operations

• Configuring Identity Manager

• Understanding the Failover Process

• Initializing the Connector State

• Failover Installation Maintenance

• When to Fail over

• Failing Over

Global Telco Deployment Information

This section describes Global Telco's existing architecture and what the company wants to achieve in this deployment. This section also lists the Identity Synchronization for Windows features that are highlighted in this case study.

Global Telco's Existing Architecture

Global Telco, a large company with 500,000 employees world-wide, is using Sun Java System Identity Manager (Identity Manager) to support users between Active Directory, Directory Server, Oracle RDBMS, Novel NDS, and other systems. The company has two main data centers: one in the United States, and one in Europe.

The company has a single Active Directory domain (gt.com) with four domain controllers, and a Sun Java System Directory Server deployment (dc=gt,dc=com) with four preferred Directory Servers and four read-only replicas.

Directory Server Information

The Sun Java System Directory Server topology includes four preferred Directory Server and four master replicas. Directory Server is the corporate directory server used to control access to web-based applications. The directory server has a single root suffix, dc=gt,dc=com. Information about users is stored in the ou=people, dc=gt,dc=example,dc=com container with uid as the naming attribute.

• Two preferred Directory Server and two master replicas are located in the United States (a separate configuration directory in the United States stores configuration information for these systems).

• Two preferred Directory Server and two master replicas are located in Europe (a separate configuration directory in Europe stores configuration information for these systems).

All four preferred Directory Server have replication agreements with each other, but the four master replicas only have replication agreements with two of the masters.

Identity Synchronization for Windows treats hub replicas the same as read-only replicas. In many scenarios, using a hub replica is preferred to using a read-only replica because a hub can be easily promoted to a preferred Directory Server.

Figure 3–1 Data Center Information for Directory Server

Active Directory Information

The Active Directory deployment has a single domain, gt.com, with two domain controllers located in the United States and two in Europe. The user information is stored in the standard cn=users container in Active Directory (cn=users,dc=gt,dc=com).

The Active Directory samaccountname attribute value matches the Directory Server uid attribute. The Active Directory domain controller with the PDC FSMO role is located in the United States office.

Figure 3–2 Data Center Information for Active Directory

Both ad1-us.gt.com and ad3-eu.gt.com are bridgehead servers that control replication between the two sites.

Global Telco's Technical Requirements

Global Telco wants to achieve the following:

• Users’ passwords for Windows systems must be synchronized with their Directory Server passwords.

• Users must be able to change passwords using native mechanisms made in either system, through the Change Password option in the Task Manager dialog box on Windows systems, and through a web-based portal for Directory Server.

  Identity Synchronization for Windows supports capturing native password changes in Directory Server and Active Directory. Users can continue to change passwords as they always have.

  ---

  Note –

  Passwords can be set in Directory Server by passing a pre-hashed password value. However, Identity Synchronization for Windows cannot synchronize passwords from Directory Server to Windows if the password is pre-hashed. Even in installations without Identity Synchronization for Windows, avoid using a pre-hashed password value because it circumvents password policy and password history.

  ---

• Existing Identity Manager functionality must be retained and continue to support users on Active Directory and Directory Server.

  Identity Synchronization for Windows requires the users’ Directory Server accounts to be explicitly linked to their Windows accounts. This linking is automatically done when Identity Synchronization for Windows is configured to synchronize creations of new users. However, because Identity Manager is provisioning both Active Directory and Directory Server accounts, Identity Synchronization for Windows will not synchronize new users. Global Telco must either run the idsync resync command periodically to link newly created users, or Identity Manager must be configured to set the necessary linking attributes when a new Directory Server entry is created.

• Support must be added for propagating native password changes made in Directory Server to all systems managed by Identity Manager.

  Identity Manager supports synchronizing Active Directory password changes to many other systems because Identity Synchronization for Windows can synchronize password changes from Directory Server to Active Directory. Integrate Identity Manager with Identity Synchronization for Windows to synchronize password changes made in Directory Server to any system that Identity Manager supports.

• High availability for failover redundancy of all services is required in the European office.

  Identity Synchronization for Windows is very robust. After all components are running, it synchronizes data without losing changes. By default, Identity Synchronization for Windows provides some high availability options, such as failover to a secondary Directory Server, and performing on-demand password synchronization against any Active Directory domain controller. It also includes a Watchdog that restarts failed processes.

  However, if the machine that runs Identity Synchronization for Windows Core or Connector experiences a hardware failure, Identity Synchronization for Windows will not synchronize users until it is reinstalled on different hardware.

  This case study addresses Global Telco's HA requirement by installing a completely separate instance of Identity Synchronization for Windows at the European office.

• All communication must use SSL and trusted certificates where possible.

  Identity Synchronization for Windows supports SSL communication for all over-the-wire communication. By default, it does not require trusted certificates for SSL communication between connectors and directory sources, but it can be configured to require trusted certificates.

Identity Synchronization for Windows Features in This Case Study

The following Identity Synchronization for Windows features are used in this case study:

• Integrating Identity Synchronization for Windows with Identity Manager to synchronize passwords

• Deploying Identity Synchronization for Windows in an HA environment with a large number of users

• Installing Identity Synchronization for Windows on multiple machines

• Establishing links between users when Identity Synchronization for Windows does not synchronize account creation

• Providing directory server access over a WAN

• Using SSL

Installation and Configuration Overview

This section provides an overview of the installation and configuration tasks for meeting Global Telco’s requirements. For details on configuring Identity Manager to coexist with Identity Synchronization for Windows, see Appendix B, Configuring Identity Manager and Identity Synchronization for Windows to Coexist.

Primary and Failover Installations

To provide a high-availability solution, Identity Synchronization for Windows must be installed in two separate locations, one in the United States and another in Europe. The deployment in the United States is the primary deployment, while the one is Europe is only meant to be used during failover scenarios.

To improve performance, the Identity Synchronization for Windows components are distributed between two machines in each location. For the deployment in the United States, the Identity Synchronization for Windows Core components are installed on config-us.gt.com, and both connectors are installed on connectors-us.gt.com. For the deployment in Europe, the Identity Synchronization for Windows Core components are installed on config-eu.gt.com, and both connectors are installed on connectors-eu.gt.com.

The primary deployment and the various communication paths are shown in the following figure. For simplicity, gt.com is dropped, and only the machine names are shown.

Figure 3–3 Primary Installation of Identity Synchronization for Windows

The Directory Server Connector and Active Directory Connector, installed on connectors-us.gt.com, communicate with each other and receive their configuration from the Message Queue that is installed with the Identity Synchronization for Windows Core.

The Active Directory Connector communicates exclusively with the ad1-us.gt.com domain controller, using LDAP. The Directory Server Connector communicates with two preferred Directory Servers. While primary Directory Server is available, the Directory Server Connector detects and propagates changes to master1-us.gt.com. If primary Directory Server is unavailable, it fails over to master2-us.gt.com to apply changes, but cannot detect further changes made at any other preferred Directory Server until master1-us.gt.com is available.

• If both master1-us.gt.com and master2-us.gt.com are unavailable, changes synchronized from Active Directory are not applied to the other preferred Directory Servers, master3-eu.gt.com and master4-eu.gt.com. Identity Synchronization for Windows treats these other preferred Directory Servers like read-only replicas, except that external changes to these preferred Directory Servers are replicated to the retro change log on master1-us.gt.com and then synchronized to Active Directory.

• The Identity Synchronization for Windows Directory Server Plug-ins running on Master1-us.gt.com and master2-us.gt.com only communicate with ad2-us, ad3-eu, and ad4-eu if ad1-us is unavailable. Similarly, the other preferred Directory Servers and replicas only communicate with master2-us.gt.com if master1-us.gt.com is unavailable.

An Identity Synchronization for Windows Directory Server Plug-in must be enabled on all eight Directory Server instances, four preferred Directory Servers and four read-only replicas.

You can enable the Directory Server Plug-in by using the following:

idsync dspluginconfig -{C/U} -D <bind DN> -w <bind pass word | -> [-h <CD hostname>] [-p <CD port no>] [-s <configuration suffix>] [-Z] [-P <cert db path>] [-m <secmod db path>] [-d <ds plugin hostname>] [-r <ds plug in port>] [-u <ds plugin user>] [-x <ds plugin user password>] [-o <database suf fix>][-q <configuration password | ->] [-W] [-B <plugin DS cert db path>] [-g <plugin DS secmod db path>]

Type idsync --help for information about the syntax.

When a directory server starts, the Directory Server Plug-in establishes a secure connection to the Directory Server Connector. After the plug-in is authenticated, the connector sends the configuration information, and the plug-in can send log messages to the central log, through the connector. The configuration includes keys for encrypting modified passwords and Active Directory information for performing on-demand password synchronization.

When a user's Active Directory password changes, Identity Synchronization for Windows sets the dspswvalidate attribute to true in the user's Directory Server entry. The user can then log in to any Directory Server, and on-demand password synchronization can originate from any server.

If the user logs in to master1-us.gt.com or master2-us.gt.com, on-demand password synchronization is performed directly to the ad1-us.gt.com Active Directory domain controller. Other domain controllers are contacted only if ad1-us.gt.com is unavailable.

If the user logs in to one of the other two preferred Directory Servers or one of the read-only replicas, on-demand password synchronization is performed against master1-us.gt.com or master2-us.gt.com. These preferred Directory Servers in turn continue the on-demand password synchronization to one of the Active Directory domain controllers.

These two actions are necessary for the following reasons:

• A read-only replica cannot update the user's Directory Server entry with the correct password if on-demand password synchronization succeeds.

• With the exception of the preferred and secondary Directory Servers, Identity Synchronization for Windows treats all Directory Server instances as read-only replicas.

After the primary installation is complete, the failover Identity Synchronization for Windows installation is performed on the two machines in Europe, config-eu.gt.com and connectors-eu.gt.com.

Figure 3–4 Failover Installation While the Primary Installation Is Active

Because the Identity Synchronization for Windows Directory Server Plug-ins have not been reinstalled, so they still receive their configuration from the Directory Server Connector running on connectors-us.gt.com, while on-demand password synchronization passes through master1-us.gt.com or master2-us.gt.com before reaching the Active Directory domain controllers.

The failover installation remains in this state until Global Telco needs to fail over to it. To complete the failover process, the Identity Synchronization for Windows Plug-in is enabled on every directory server, which changes its startup configuration to communicate with the Directory Server Connector running on connectors-eu.gt.com.

Figure 3–5 Primary Installation After Reinstalling the Identity Synchronization for Windows Plug-ins

Setting up the failover installation significantly increases the amount of time required to deploy Identity Synchronization for Windows. However, this up-front cost is offset by having the ability to quickly fail over to the alternate deployment if necessary.

Periodically Linking New Users

Identity Synchronization for Windows is not used to synchronize the creation of new users. Therefore, the idsync resync command is executed periodically to establish links between newly created users. An LDAP filter is passed to this command to resynchronize only the subset of users that have been created since the command was last executed.

See Periodic idsync resync Operations for more information.

Large Deployment Considerations

Due to the large size of its deployment, Global Telco takes the following steps to increase the performance of Identity Synchronization for Windows.

• Identity Synchronization for Windows components are distributed across two dedicated machines.

• The log level is set to INFO except when diagnosing transient problems.

• Message Queue Broker memory limits are increased (see the Sun Java System Directory Server Enterprise Edition 6.0 Installation Guide for more information).

• The default number of worker threads in the connectors is increased.

• The idsync resync command is run over batches of users to reduce the peak load on Message Queue.

  Each of these actions is discussed in detail in the next section.

Configuration Walkthrough

This section provides high-level steps to configure Identity Synchronization for Windows in a high-availability environment.

Only important steps are provided. Any configuration instructions already discussed in the Example Bank case study have been omitted.

For detailed configuration instructions, see the Sun Java System Directory Server Enterprise Edition 6.0 Installation Guide.

Primary Installation

After the Identity Synchronization for Windows Core is installed on config-us.gt.com, the Identity Synchronization for Windows Console is started. You configure the preferred Directory Server source first.

In this case, master1-us.gt.com is chosen as the preferred Directory Server. The connector communicates with the Directory Server source over SSL.

master2-us.gt.com is chosen as the secondary Directory Server. The connector communicates with Directory Server over SSL.

Because Global Telco requires the strictest security possible, the Directory Server Connector will require a trusted SSL certificate from the directory server, and the Identity Synchronization for Windows Directory Server Plug-ins will communicate over SSL to Active Directory.

Note that the Identity Synchronization for Windows Plug-ins inherit the SSL configuration of the directory server. Therefore, if the Directory Server requires trusted certificates, the plug-in can only communicate with Active Directory if it provides a trusted certificate. Enabling these enhanced security options implies the following additional installation actions.

ad1-us.gt.com is the PDC FSMO role owner and is selected as the domain with which the controller for the Active Directory Connector will communicate. The connector communicates over SSL.

All three remaining domain controllers will be used for failover during on-demand password synchronization.

Because Global Telco requires the strictest security possible, the Active Directory Connector will require a trusted SSL certificate from ad1-us.gt.com.

Enabling this advanced security option implies additional installation steps as outlined below.

The only default global setting that is changed is the synchronization of attribute modifications from Active Directory to Directory Server, and from Directory Server to Active Directory.

Only passwords are synchronized. No additional attributes are synchronized.

A single SUL, GT_USERS, is created as shown in Primary Installation.

Active Directory users are stored under the default cn=users,dc=gt,dc=com container. The existing users (Administrator, Guest, TsInternetUser, and iswUser) are excluded from synchronization.

The Directory Server users are stored in the default ou=people,dc=gt,dc=com container.

After the configuration is saved, each connector is installed on connectors-us.gt.com, and the Identity Synchronization for Windows Plug-in is installed.

				  bash-2.05# ./idsync printstat -w <password omitted\> -q <password omitted\>
				  Exploring status of connectors, please wait...

				  Connector ID: CNN100
				     Type: Sun Java(TM) System Directory
				     Manages: dc=gt,dc=com (ldaps://master1-us.gt.com:636)
				    (ldaps://master2-us.gt.com:636)
				     State: READY
				     Installed on: connectors-us.gt.com
				     Plugin SUBC100 is installed on ldaps://master1-us.gt.com:636
				     Plugin SUBC101 is installed on ldaps://master2-us.gt.com:636
				     Plugin SUBC102 is installed on ldaps://master3-eu.gt.com:636
				     Plugin SUBC103 is installed on ldaps://master4-eu.gt.com:636
				     Plugin SUBC104 is installed on ldaps://replica1-us.gt.com:636
				     Plugin SUBC105 is installed on ldaps://replica2-us.gt.com:636
				     Plugin SUBC106 is installed on ldaps://replica3-eu.gt.com:636
				     Plugin SUBC107 is installed on ldaps://replica4-eu.gt.com:636

				  Connector ID: CNN101
				     Type: Active Directory
				     Manages: gt.com (ldaps://ad2-us.gt.com:636) (ldaps://ad3-eu.gt.com:636)
				    (ldaps://ad4-eu.gt.com:636) (ldaps://ad1-us.gt.com:636)
				     State: READY
				     Installed on: connectors-us.gt.com

				  Sun Java(TM) System Message Queue Status: Started

				  Checking the System Manager status over the Sun Java(TM) System Message Queue.

				  System Manager Status: Started

				  Remaining Installation and Configuration Steps:

				  1. Install the Sun Directory Server Plugin on every other master 
				  and read-only replica that manage users under dc=gt,dc=com.
				  2. Run 'idsync resync' to establish links between existing Directory Server 
				  and Windows users.
				  3. Start synchronization using the console or the 'idsync startsync' command.
				  SUCCESS

Failover Installation

After the primary installation is complete, you install the Identity Synchronization for Windows Core on config-eu.gt.com, and configure it using the console.

master3-eu.gt.com is the preferred Directory Server in the failover installation.

master4-eu.gt.com is the secondary Directory Server in the failover installation.

ad3-eu.gt.com is chosen as the domain controller with which the Active Directory Connector will communicate.

Note that a warning will be displayed stating that the password updates might become slow because ad3-eu.gt.com is not the PDC FSMO role owner. This warning can be ignored because changing the PDC FSMO role to this domain controller is part of the failover procedure. A similar warning is also displayed when the configuration is saved.

The remaining domain controllers are selected for failover during on-demand password synchronization.

bash-2.05# /opt/SUNWisw/bin/idsync printstat -q <
omitted password\> -w <omitted password\>
Exploring status of connectors, please wait...
Connector ID: CNN100
Type: Sun Java(TM) System Directory
Manages: dc=gt,dc=com (ldaps://master3-eu.gt.com:636)
(ldaps://master4-eu.gt.com:636)
State: READY
Installed on: connectors-eu.gt.com

Connector ID: CNN101
Type: Active Directory
Manages: gt.com (ldaps://ad1-us.gt.com:636) (ldaps://ad2-us.gt.com:636)
(ldaps://ad4-eu.gt.com:636) (ldaps://ad3-eu.gt.com:636)
State: READY
Installed on: connectors-eu.gt.com

Sun Java(TM) System Message Queue Status: Started

Checking the System Manager status over the Sun Java(TM) System Message Queue.

System Manager Status: Started

Remaining Installation and Configuration Steps:

1. Install the Sun Directory Server Plugin at master ldaps://master3-eu.gt.com:636 
by re-running the installer.
2. Install the Sun Directory Server Plugin at master ldaps://master4-eu.gt.com:636 
by re-running the installer.
3. Install the Sun Directory Server Plugin on every other master and read-only replica 
that manage users under dc=gt,dc=com.
4. Run 'idsync resync' to establish links between existing Directory Server 
and Windows users.
5. Start synchronization using the console or the 'idsync startsync' command.
SUCCESS

Setting Up SSL

Global Telco requires that all network traffic is encrypted, so SSL is used with trusted certificates for all LDAP connections. This setup includes connections between the following:

• Directory Server Connectors and the preferred and secondary Directory Servers

• Active Directory Connector and the Active Directory domain controller

• Preferred and secondary Directory Servers, and the Active Directory domain controllers for on-demand password synchronization

• All other preferred and secondary Directory Servers and read-only replicas, and the preferred and secondary Directory Servers for on-demand password synchronization.

  Companies typically use only a few Certificate Authorities (CA) to sign certificates, so the simplest approach to setting up SSL for Identity Synchronization for Windows is to add all CA certificates to every component’s certificate database. Global Telco uses two Certificate Authorities: one to sign its Directory Server certificates and one to sign its Active Directory certificates. Both of these CA certificates are added to the certificate database of each connector and each directory server instance.

The idsync certinfo command displays the steps for configuring SSL for Identity Synchronization for Windows components, based on the current configuration. The command does not have access to each component’s certificate database, so it cannot determine if the steps have already been followed.

The output of this command is shown for the following primary installation. The output for the failover installation is identical except that the roles of the U.S. and European systems are reversed.

bash-2.05# /opt/SUNWisw/bin/idsync certinfo -q <omitted password\> -w <omitted password\>
Connector: CNN100
Installation Host: connectors-us
Installation Path: /opt
Certificate Database Location:   /var/opt/SUNWisw/etc/CNN100
**The Directory Server Connector's certificate database must contain the 
CA certificate used to sign Directory Server's SSL certificate. If this 
certificate has not already been added to the connector's certificate 
database, please export the CA certificate and import into Directory Server 
Connector certificate database for server ldaps://master1-us.gt.com:636.


**The Directory Server's certificate database must contain the CA certificate 
used to sign the Active Directory's SSL certificate. If this certificate has 
not already been added to the Directory Server's certificate database, please 
export the CA certificate from the Active Directory at ldaps://ad1-us.gt.com:636 
and import into Directory Server certificate database for server 
ldaps://master1-us.gt.com:636.


**The Directory Server's certificate database must contain the CA certificate 
used to sign the Active Directory's SSL certificate. If this certificate has 
not already been added to the Directory Server's certificate database, please 
export the CA certificate from the Active Directory at ldaps://ad2-us.gt.com:636 
and import into Directory Server certificate database for server 
ldaps://master1-us.gt.com:636.


**The Directory Server's certificate database must contain the CA certificate 
used to sign the Active Directory's SSL certificate. If this certificate has 
not already been added to the Directory Server's certificate database, please 
export the CA certificate from the Active Directory at ldaps://ad3-eu.gt.com:636 
and import into Directory Server certificate database for server 
ldaps://master1-us.gt.com:636.


**The Directory Server Connector's certificate database must contain the 
CA certificate used to sign the Directory Server's SSL certificate. 
If this certificate has not already been added to the connector's certificate 
database, please export the CA certificate and import into Directory Server 
Connector certificate database for server ldaps://master2-us.gt.com:636.

**The Directory Server's certificate database must contain the 
CA certificate used to sign the Active Directory's SSL certificate. 
If this certificate has not already been added to the Directory Server's 
certificate database, please export the CA certificate from the 
Active Directory at ldaps://ad1-us.gt.com:636 and import into 
Directory Server certificate database for server 
ldaps://master2-us.gt.com:636.


**The Directory Server's certificate database must contain the CA certificate 
used to sign the Active Directory's SSL certificate. If this certificate has 
not already been added to the  Directory Server's certificate database, please 
export the CA certificate from the Active Directory at 
ldaps://ad2-us.gt.com:636 and import into Directory Server certificate 
database for server ldaps://master2-us.gt.com:636.


**The Directory Server's certificate database must contain the CA certificate 
used to sign the Active Directory's SSL certificate. If this certificate has 
not already been added to the Directory Server's certificate database, please 
export the CA certificate from the Active Directory at 
ldaps://ad4-eu.gt.com:636 and import into Directory Server certificate 
database for server ldaps://master1-us.gt.com:636.


**The Directory Server's certificate database must contain the CA certificate 
used to sign the Active Directory's SSL certificate. If this certificate has not 
already been added to the Directory Server's certificate database, please 
export the CA certificate from the Active Directory at ldaps://ad3-eu.gt.com:636 
and import into Directory Server certificate database for server 
ldaps://master2-us.gt.com:636.


**The Directory Server's certificate database must contain the CA certificate 
used to sign the Active Directory's SSL certificate. If this certificate has not 
already been added to the Directory Server's certificate database, please export 
the CA certificate from the Active Directory at ldaps://ad4-eu.gt.com:636 and 
import into Directory Server certificate database for server 
ldaps://master2-us.gt.com:636.


Connector: CNN101
Installation Host: connectors-us
Installation Path: /opt
Certificate Database Location: /var/opt/SUNWisw/etc/CNN101


**The Active Directory Connector's certificate database must contain the CA 
certificate used to sign the Active Directory's SSL certificate. If this certificate 
has not already been added to the Active Directory Connector certificate database, 
please export the CA certificate from the Active Directory and import into 
Active Directory Connector's certificate database for server 
ldaps://ad1-us.gt.com:636.

**The Active Directory Connector's certificate database must contain 
the CA certificate used to sign the Active Directory's SSL certificate. If this 
certificate has not already been added to the Active Directory Connector 
certificate database, please export the CA certificate from the Active Directory 
and import into Active Directory Connector's certificate database for server 
ldaps://ad2-us.gt.com:636.


**The Active Directory Connector's certificate database must contain the 
CA certificate used to sign the Active Directory's SSL certificate. 
If this certificate has not already been added to the Active Directory Connector 
certificate database, please export the CA certificate from the Active Directory 
and import into Active Directory Connector's certificate database for server 
ldaps://ad3-eu.gt.com:636.


**The Active Directory Connector's certificate database must contain the 
CA certificate used to sign the Active Directory's SSL certificate. If this 
certificate has not already been added to the Active Directory Connector certificate 
database, please export the CA certificate from the Active Directory and import into 
Active Directory Connector's certificate database for server 
ldaps://ad4-eu.gt.com:636.


SUCCESS

Setting Up SSL summarizes SSL communication between components in this installation, including trust requirements for the primary and failover installations.

The following table provides more detailed information.

In this installation, Global Telco adds both of the CA certificates to the certificate databases of the four connectors and the eight directory servers.

See the Sun Java System Directory Server Enterprise Edition 6.0 Installation Guide for detailed instructions on adding certificates to the certificate databases. The Directory Server and connectors must be restarted after the certificates have been added. The Directory Server must be restarted after the Identity Synchronization for Windows Plug-in is installed. Therefore, add the CA certificates to the Directory Servers' certificate databases before the Identity Synchronization for Windows Plug-in is installed.

Increasing Connector Worker Threads

By default each Directory Server and Active Directory connector uses four worker threads to apply changes. This number is increased in the Global Telco deployment to improve connector performance, especially during idsync resync operations. The number of connector threads is stored in the configuration directory in the pswNumOutboundConnectorThreads attribute, which is in the pswSunDirectoryGlobals entry and the pswActiveDirectoryGlobals entry. Before manually editing the configuration, all Identity Synchronization for Windows consoles must be closed.

• To find the pswSunDirectoryGlobals entry, the following command is used:

  bash-2.05# ./ldapsearch -b "ou=identitysynchronization,ou=services, dc=gt,dc=com" -D "cn=Directory Manager" -w <omitted password\> "(&(objectclass=pswSunDirectoryGlobals) (pswversion\>=0))" pswNumOutboundConnectorThreads

  dn: cn=136,ou=Sun,ou=Globals,cn=active[13],ou=GlobalConfig,ou=1.1, 
  ou=IdentityS ynchronization,ou=Services,dc=gt,dc=com
  pswNumOutboundConnectorThreads: 4

  The entry that must be modified is shown here:

  cn=136,ou=Sun,ou=Globals,cn=active[13],ou=GlobalConfig,ou=1.1,
  ou=IdentitySynchronization,ou=Services,dc=gt,dc=com.

• To find the pswActiveDirectoryGlobals entry, the following command is used:

  bash-2.05# ./ldapsearch -b "ou=identitysynchronization,ou=services, dc=gt,dc=com" -D "cn=Directory Manager" -w <omittied password\> "(&(objectclass=pswActiveDirectoryGlobals)(pswversion\>=0))" pswNumOutboundConnectorThreads

  dn: cn=110,ou=ActiveDirectory,ou=Globals,cn=active[13],
  ou=GlobalConfig, ou=1.1, 
  ou=IdentitySynchronization,ou=Services,dc=gt,dc=com
  pswNumOutboundConnectorThreads: 4

  The entry that must be modified is shown here:

  cn=110,ou=ActiveDirectory,ou=Globals, cn=active[13],ou=GlobalConfig,ou=1.1,
  ou=IdentitySynchronization,ou=Services,dc=gt,dc=com.

These two entries are modified to increase the number of threads to a maximum of 20:

bash-2.05# ./ldapmodify -D "cn=Directory Manager" -w <omitted password\>

dn: cn=136,ou=Sun,ou=Globals,cn=active[13],ou=GlobalConfig,ou=1.1, 
ou=IdentitySynchronization,ou=Services,dc=gt,dc=com
changetype: modify
replace: pswNumOutboundConnectorThreads
pswNumOutboundConnectorThreads: 20

modifying entry cn=136,ou=Sun,ou=Globals,cn=active[13],ou=GlobalConfig, ou=1.1,
ou=IdentitySynchronization,ou=Services,dc=gt,dc=com
dn: cn=110,ou=ActiveDirectory,ou=Globals,cn=active[13],ou=GlobalConfig, ou=1.1,
ou=IdentitySynchronization,ou=Services,dc=gt,dc=com
changetype: modify
replace: pswNumOutboundConnectorThreads
pswNumOutboundConnectorThreads: 20
modifying entry cn=110,ou=ActiveDirectory,ou=Globals,cn=active[13], 
ou=GlobalConfig,ou=1.1,ou=IdentitySynchronization,ou=Services,dc=gt,dc=com

After these values are changed, the Identity Synchronization for Windows daemon on the system with the Identity Synchronization for Windows Core, is restarted to notify the System Manager to pick up the new configuration.

Increasing the number of connector threads also increases the maximum number of LDAP connections that the connector will keep open to the directory.

Aligning Primary and Failover Configurations

Even though the primary and failover installations have similar configurations, some generated configuration parameters differ between the two deployments:

• The password used by the Directory Server Connector's uid=PSWConnector,dc=gt,dc=com user

  Although this user is created when Directory Server is “prepared” for synchronization, the password is set when the Directory Server Connector is installed. The randomly generated password is set in the directory server entry and then stored in the configuration.

• The encryption key used by the Identity Synchronization for Windows Plug-in to encrypt passwords in the retro changelog

  This key is randomly generated when the configuration is first saved.

Both of these values are encrypted and stored in the configuration directory with the rest of the Identity Synchronization for Windows configuration. However, the values cannot be copied between the two configurations because the encrypted values are unique to each deployment.

The limitation for the uid=PSWConnector entry has a workaround because Directory Server allows an entry to have multiple password values. During the installation process, the uid=PSWConnector entry can be manually modified to store the password used in the primary configuration and the password used in the failover configuration.

However, because the same encryption key cannot be used for both configurations, some password changes might be lost during failover. The failover process includes reinstalling the Identity Synchronization for Windows Plug-ins on each directory server so that they receive their configuration from the failover installation instead of the primary installation. Any password change made in Directory Server during this period will be lost. Identity Synchronization for Windows will log a message about the lost password.

Setting Multiple Passwords for uid=PSWConnector

After installing the Directory Server Connector for the primary installation, but before installing the Directory Server Connector for the failover installation, the password for the uid=PSWConnector user must be retrieved and saved:

bash-2.05# ./ldapsearch -h master1-us -b "dc=gt,dc=com" -D "cn=Directory Manager" 
-w <omitted password\> "(uid=PSWconnector)" userpassword
version: 1
dn: uid=PSWConnector,dc=gt,dc=com
userpassword: {SSHA}OUYr10Y2mHIyZfyVLM4O0nYi4UZGNSAVlAERRg==

{SSHA}OUYr10Y2mHIyZfyVLM4O0nYi4UZGNSAVlAERRg== is the password that the primary Directory Server Connector uses to connect to the directory server. Installing the Directory Server Connector for the failover installation overwrites this password. At this point, retrieve the entry again:

bash-2.05# ./ldapsearch -h master1-us -b "dc=gt,dc=com" -D "cn=Directory Manager" 
-w <omitted password\> "(uid=PSWconnector)" userpassword
version: 1
dn: uid=PSWConnector,dc=gt,dc=com
userpassword: {SSHA}k9AFSUGsY1NK038PvIB4lJzVNb0sQHh4JHJXFQ==

{SSHA}k9AFSUGsY1NK038PvIB4lJzVNb0sQHh4JHJXFQ== is the password that the failover Directory Server Connector uses to connect to the Directory Server. At this point, the Directory Server Connector for the primary installation can no longer log in to the directory, so modify the entry to include both passwords.

bash-2.05# ./ldapmodify -h master1-us -D "cn=Directory Manager" 
-w <omitted password\>
dn: uid=PSWConnector,dc=gt,dc=com
changetype: modify
replace: userpassword
userpassword: {SSHA}OUYr10Y2mHIyZfyVLM4O0nYi4UZGNSAVlAERRg==
userpassword: {SSHA}k9AFSUGsY1NK038PvIB4lJzVNb0sQHh4JHJXFQ==
modifying entry uid=PSWConnector,dc=gt,dc=com

After this process is complete, both Directory Server Connectors will be able to log in to the directory. To verify this, stop and restart the Identity Synchronization for Windows daemon for the primary installation on connectors-us.gt.com, and for the failover installation on connectors-us.gt.com. After the connectors start and receive their configuration, they will open a connection to the directory. If there are any problems with the credentials, they are reported in the central logs.

Every time the Directory Server Connector is installed, a new password is generated and written to the uid=PSWConnector entry. If Directory Server Connector is uninstalled and reinstalled, this procedure must be followed again. Also, if the Directory Server Connector for the failover installation was installed before the primary uid=PSWConnector password was retrieved, save the current uid=PSWConnector password (for the failover configuration), uninstall and reinstall the primary Directory Server Connector, and then retrieve the current uid=PSWConnector password (for the primary configuration).

Initial idsync resync Operations

The next step in the installation and configuration process is to run idsync resync. In this deployment, idsync resync takes these actions:

• Establishes links between existing Active Directory and Directory Server users

• Resets the Directory Server password to the Active Directory password by forcing on-demand password synchronization the first time that a user logs into Directory Server.

• Primes the Active Directory Connector's object cache so that changes to existing users are not falsely interpreted as newly created users

Performing a full idsync resync operation is expensive in Identity Synchronization for Windows. For a deployment as large at gt.com (500,000 users), the operation might take a few hours to complete. It also places a heavy load on Message Queue because all users and log messages are sent over Message Queue during the idsync resync operation. To reduce this load, do the following:

1. Increase the Message Queue Broker available memory to 1 Gbyte as described in the Sun Java System Directory Server Enterprise Edition 6.0 Installation Guide.

2. Change the Identity Synchronization for Windows system-wide log level from the default FINE to INFO, using the Console.

   ---

   Note –

   If you need to increase the log level to debug an issue, use the -a option of idsync resync to restrict the operation to only the affected users.

   ---

3. Divide the 500,000 users into five separate idsync resync batches by using the -a option.

   This process is described in more detail in the next section.

These steps are also performed for the failover installation.

Initial idsync resync Operation for Primary Installation

The first time that you run idsync resync, divide the Active Directory users into five batches to reduce the peak load on Message Queue. Use the -a <ldap-filter\> option with idsync resync to limit the scope of the idsync resync operation to a subset of Active Directory users. When dividing users, the following must be ensured:

• The union of the LDAP filters includes every user that should be synchronized.

  For example, all characters might not be English or even ASCII, so resynchronization based on the 26 letters of the English alphabet will skip users with non-English characters.

• The LDAP filters can overlap but a significant overlap will increase the total running time of the operation.

• Batch sizes are not too large, to keep the Message Queue Broker reaching its memory limits.

• The attributes in the filter are indexed in Active Directory.

  Consult Microsoft's documentation to determine which Active Directory attributes are indexed. The standard set of indexed attributes includes cn, sn, and samaccountname.

• Certain types of searches on indexed attributes can be expensive.

  For example, an absence search, (!(sn=*)), will result in a time-out error even though it is indexed.

• If the filter includes attributes that are not mandatory, for example, sn, all entries must have a value for that attribute so that entries are not overlooked.

• If a user attribute used in the filter can change value during the operation, users whose attributes change might not be synchronized.

  For example, if an entry’s sn value changes from Smith to Anderson after the first batch of entries has been synchronized, this user will be skipped. The following script exploits Active Directory's uSNChanged attribute to resynchronize all entries that changed during the sequence of idsync resync operations.

Running idsync resync in batches will not significantly increase the total time required to do a full idsync resync operation. It might even speed the operation because it prevents the Message Queue Broker from thrashing as it reaches its memory limits.

Only a single idsync resync operation can be run at a time, not in parallel.

The -a <ldap-filter\> option always applies to the source of the idsync resync operation. When users are linked with the -f option, Active Directory must be the source of the operation. Thus, only consider how to divide users based on the available Active Directory attributes.

Due to these requirements, Global Telco is resynchronizing its users based on each user's cn attribute, a mandatory attribute that is indexed. Users are categorized into the following groups:

• cn <= a

• a <= cn <= d

• d <= cn <= h

• h <= cn <= m

• m <= cn <= s

• s <= cn

If a user's cn exactly matches one of the boundary conditions (for example, cn=d), the user will be synchronized twice, introducing negligible overhead but not causing any problems.

Global Telco then uses the following Perl script to iterate through the idsync resync operations. The resync-batch.pl script first retrieves the current maximum uSNChanged value from Active Directory. The specified Active Directory host must be the same host as the one the Active Directory Connector uses for communication because uSNChanged values differ between Active Directory domain controllers. The script then iterates through each idsync resync operation, pausing for one minute between the operations to allow the connectors to reset and settle. The last search filter matches all user entries that changed during the other idsync resync operations.

#! /usr/bin/perl
#
# This script executes a sequence of resync operations.
# Each resync operation occurs on a subset of users
# based on an LDAP filter.
#
# Parameters specific to this deployment
#
my $linkFile = "/var/opt/SUNWisw/samples/linkusers-simple-gt.cfg";
my $idsync = "/opt/SUNWisw/bin/idsync";
my $adHost = "ad1-us.gt.com";
my $secBetweenOps = 60;
my $dsPassword = "<omitted password\>";
 my $configPassword = "<omitted password";

# Guard against users whose cn changes during this operation
# by retrieving the maximum uSNChanged value before we start
# the operation, and later running resync for only these users.
my $origUsn = ”ldapsearch -h $adHost -b ’’ -s base ’(objectclass=*)’ \\
highestCommittedUSN | grep highestCommittedUSN | sed ’s/[^0-9]//g’”;
chomp $origUsn;
# Run each user in a batch
my @BATCH_FILTERS =  ("
            (cn<=a)",             # cn <= a
            "(&(cn\>=a)(cn<=d))", # a <= cn <= d
            "(&(cn\>=d)(cn<=h))", # d <= cn <= h
            "(&(cn\>=h)(cn<=m))", # h <= cn <= m
            "(&(cn\>=m)(cn<=s))", # m <= cn <= s
            "
            (cn\>=s)",            # s <= cn
            "(uSNChanged\>=$origUsn)",# modified users);
# Run each user in a batch
for my $filter (@BATCH_FILTERS) {
       print "Running resync with filter ’$filter’.\\n";
       my $command = "$idsync resync -a ’$filter’ -f $linkFile"."
       -i NEW_LINKED_USERS -q $configPassword -w $dsPassword";
       system($command);
       print "Waiting $secBetweenOps before next iteration.\\n";
       sleep $secBetweenOps;
}

Notice that the -i NEW_LINKED_USERS option is passed to idsync resync, which resets all Directory Server passwords to match their Active Directory counterparts after the entries are linked.

If Global Telco did not want to reset the passwords for the Directory Server accounts, it would not run this command with the -k option, which only links users. The idsync resync command primes the object cache database in the Active Directory Connector, which is used to detect changes to entries.

When idsync resync is run with the -k option, the object cache is not primed, and if the connector detects a change to an existing user, it will erroneously assumes that the change is a new user. Thus, if you run idsync resync -k, you must run the idsync resync command again with the -u option, which updates the object cache only.

Initial idsync resync Operation for Failover Installation

You must also run the idsync resync command for the failover installation. The links between the entries are already established, and idsync resync is only needed to prime the object cache database for the Active Directory Connector. Therefore, the command is run with the -u option.

To avoid overloading Message Queue with log messages, the following must be done:

• Increase the Message Queue Broker maximum memory

• Change the Identity Synchronization for Windows log level to INFO

• Run idsync resync over the entire user population in batches

You must modify the script in the previous section and run it on the failover installation Core, config-eu.gt.com. Change the parameters at the top of the script, and replace the -i NEW_LINKED_USERS option with just -u.

Periodic idsync resync Operations

Periodic idsync resync Operation for Primary Installation

As Global Telco uses Identity Manager to provision users and has not configured Identity Synchronization for Windows to synchronize the creation of new users, idsync resync must be run periodically to establish links between recently created users. If these users are not linked, modifications to their passwords will not synchronize. Running a full idsync resync operation for 500,000 users could take a few hours because synchronization of new changes is delayed during an idsync resync operation. It is unacceptable to have Identity Synchronization for Windows offline for so long, so Global Telco uses the Active Directory usnCreated attribute to synchronize only those users that have been created recently. The administrator uses the following resync-recent.pl script, which is run every hour, to synchronize users that have been created in the last three days. This script is run on users created in the last three days instead of only the last hour because a new employee might not have a Directory Server account until a few days after the Active Directory account is created.

The script is run periodically with the following arguments, using cron:

resync-recent.pl ad1-us.gt.com usnCreated 
-f /var/opt/SUNWisw/samples/linkusers-simple-gt.cfg 
-i NEW_LINKED_USERS -b -w - -q - < /var/opt/SUNWisw/passwords

The first argument is the name of the Active Directory domain controller that the connector communicates with. The second argument determines whether the idsync resync command should be run only on entries that were created recently. The remaining arguments are passed directly to resync. In this case, the Directory Server password is reset for all users that are newly linked. The - value used for the two password options directs the idsync resync command to read the values from STDIN; which prevents passwords from appearing in the command line and being available to anyone on the system using commands such as ps. The configuration directory password and the configuration password are written to /var/opt/SUNWisw/passwords, and this file is then protected with the strictest file system permissions.

resync-recent.pl script:
#!/usr/bin/perl

# This sample script shows how to run idsync resync only on users
# that have changed recently and can be used to reduce the
# impact of running resync frequently.  By default it
# only resync's users that were modified or created in
# the last three days.  The script stores a daily history
# of Active Directory's highestCommittedUSN attribute.
#
# The arguments of the command are
#
#  <Active Directory-domain-controller-name\>
#  (usnChanged|usnCreated)
#  [args-for-resync]
#
# To link users that were created recently run the command
#  ad.example.com usnCreated -k -f link.cfg -q <pw\> -q <pw\>
#
# To synchronize all users that were modified recently
#  ad.example.com usnChanged -q <pw\> -q <pw\>
#
# To prime the object cache for users that were modified recently
#  ad.example.com usnChanged -u -q <pw\> -q <pw\>
#
# NOTE: this script is only provided as a guide.
# It must be adapted to the specific Identity
# Synchronization for Windows environment by
# changing the paths and options below as appropriate
# and adding additional error handling code.
#
my $USAGE = "USAGE: resync-recent.pl “.
    “<Active Directory-domain-controller-name\> ".
    "(usnChanged|usnCreated) [args-for-resync]\\n";

my $IDSYNC = "/opt/SUNWisw/bin/idsync";
my $USN_FILE = "/opt/SUNWisw/bin/usnHistory";
my $MAX_DAYS_HISTORY = 3;

         

#
# SCRIPT BEGIN
#

#
# Argument parsing
#
my $adDomainController = shift @ARGV or die "$USAGE";
my $usnSearchAttr = shift @ARGV or die "$USAGE";
$usnSearchAttr =~ /usnChanged/i or
     $usnSearchAttr =~ /usnCreated/i or
    die "$USAGE\\n";
my $resyncArgs = getArgsAsString(@ARGV);

#
# Read the highestCommittedUSN history and add today's
# date to it if not it's there and then write the history
# out.
#
my %usnHistory = readUsnHistory();
my $today = getCurrentDate();
if (!$usnHistory{$today}) {
    $usnHistory{$today} = getHighestUsn($adDomainController);
}
writeUsnHistory(%usnHistory);

#
# Run the resync command based on the oldest usnChanged
# value in the history.
#
my $oldestUsn = getOldestUsn(%usnHistory);
my $filter = "($usnSearchAttr\>=$oldestUsn)";
my $resyncCmd = "$IDSYNC resync -a \\"$filter\\" $resyncArgs";
print "Running $resyncCmd\\n";
system($resyncCmd);

#
# SCRIPT END
#

#
# Return the current date as a string, e.g. 2004/03/04
#
sub getCurrentDate {
    my ($day, $month, $year) = (localtime(time))[3,4,5];
    $month++;
    $year += 1900;
    return sprintf "%d/%02d/%02d", $year, $month, $day;
}
#
# Searches the root DSE at the specified host and returns
# the highestCommittedUSN value.
#
sub getHighestUsn {
    my $adHost = shift @_;
    my $cmd = "ldapsearch -h $adHost -b \\"\\" -s base ".
        "\\"(objectclass=*)\\" highestCommittedUSN | ".
        "grep highestCommittedUSN | sed \\"s/[^0-9]//g\\"";
    my $highestUsn = '$cmd';
    chomp $highestUsn;
    print "highestCommittedUSN at $adHost is $highestUsn.\\n";
    return $highestUsn;
}

#
# Converts the command line args into a string.
#
sub getArgsAsString {
    my $args = "";
    for my $arg (@_) {
        $args .= " \\"$arg\\"";
    }
    return $args;
}

#
# Returns the oldest usnChanged value from the history.
#
sub getOldestUsn {
    my %usnHistory = @_;

    my @dates = getHistoryDates(%usnHistory);

        # Return the last element.
    return $usnHistory{$dates[$#dates]};
}

#
# Return a sorted list of the dates in the history.
#
sub getHistoryDates {
    my %history = @_;
    return reverse sort(keys %usnHistory);
}

#
# Writes the most recent daily history of highestCommittedUSN.
# No more than $MAX_DAYS_HISTORY days history is recorded.
#
sub writeUsnHistory {
    my %usnHistory = @_;
    if (!open(OUT, "\>$USN_FILE")) {
        print STDERR "Could not open $USN_FILE for writing.\\n";
        return;
    }

        # Write the history up to the last $MAX_DAYS_HISTORY days.
    my @dates = getHistoryDates(%usnHistory);
    for (my $i = 0;
         $i <= $#dates and $i < $MAX_DAYS_HISTORY;
         $i++)
     {
        my $date = $dates[$i];
        print OUT "$date:$usnHistory{$date}\\n";
            }

    close(OUT);
}

#
# Reads the daily history of highestCommittedUSN
#

sub readUsnHistory {
    my %usnHistory = ();
    if (!open(IN, "<$USN_FILE")) {
        print STDERR "Could not read history from $USN_FILE.\\n";
        return %usnHistory;
    }

    while (my $line = <IN\>) {
        chomp $line;
        my ($date, $usn) = split /:/, $line;
        $usnHistory{$date} = $usn;
    }
    close(IN);

    return %usnHistory
}

An alternative to running idsync resync periodically is to modify the process to set the necessary link information when the Directory Server entry is processed. The process is straightforward:

1. Add the dspswuser object class to the user entry.

2. Set the dspswuserlink attribute to match the user's objectguid attribute (from Active Directory).

3. (Optional) Set the dspswvalidate attribute to true to force on-demand password synchronization.

Periodic idsync resync Operation for Failover Installation

To speed the failover process, the idsync resync operation is run periodically on the failover installation to keep the object cache database in the Active Directory Connector up-to-date. If the object cache is not kept up-to-date, the Active Directory Connector will detect and propagate many changes that were already synchronized by the primary installation. Not keeping the object cache database up-to-date will also significantly increase the load, and place a heavier load on Directory Server during the failover scenario.

The same resync-recent.pl script that is used in the primary installation is used in this installation except that it is run from the system that contains the failover installation Core, config-eu.gt.com. The cron command is used to run the script daily with the following arguments. The -u option is specified to update only the Active Directory Connector's object cache.

resync-recent.pl ad3-eu.gt.com usnCreated -u -b -w - -q - < /var/opt/SUNWisw/passwords

The more often this script is run in the failover environment, the more likely changes will be lost during the failover process. idsync resync -u should not be run after the primary installation fails. If the command is run often (for example, every hour), it is likely that it will be run while the primary installation has failed, but the failure has not yet been detected. As this script keeps track of a three-day history of the highestCommittedUSN values, it could be updated to search for entries that were modified in the last three days but not modified in the last day. As long as the primary installation failure is detected within one day and the cron job of this script was stopped, no Active Directory changes are lost.

Configuring Identity Manager

For details on configuring the Sun Java System Identity Manager to coexist with Identity Synchronization for Windows, see Appendix B, Configuring Identity Manager and Identity Synchronization for Windows to Coexist.

Understanding the Failover Process

The goal of the failover process is to avoid losing any changes that occurred between the time the primary system went down and the failover system was brought up. If changes are lost, Identity Synchronization for Windows should at least report the changes that are lost.

After synchronization is started for the first time, an Identity Synchronization for Windows Connector can continue to detect changes that occur in these situations:

• When synchronization is stopped

• When the connector process is not running, for example, during a system restart

• When the network connection between the connector and the directory source is down

• During a idsync resync operation

The Directory Server and Active Directory connectors use similar mechanisms to achieve this goal.

Directory Server Connector

The Directory Server Connector persistently stores the changenumber of the last retro changelog entry that it has processed. When the connector begins detecting changes again, it processes all changes that occurred when it was not running, starting with the last changenumber that it processed. This changenumber is stored in the connector's persist directory, for example, /var/opt/SUNWisw/persist/ADP100/accessor.state. If this file does not exist, the connector detects only new changes in Directory Server.

This way of processing the changes has an effect on the failover process. If synchronization is never started for the failover installation, the file will not exist, and the Directory Server Connector will only detect new changes. Changes that occurred during the failover process are lost. However, initializing the accessor.state file presents its own problems. If synchronization is started immediately after installation to produce the accessor.state file, and then stopped, when synchronization is started after failing over, the Directory Server Connector will try to process all retro changelog entries. To strike a balance between these two options, the retro changelog Plug-in on the failover installation's preferred master (master3-eu.gt.com) is configured to only store changes for one day (this limit is increased during the failover process). When synchronization is started for the failover installation, the Directory Server Connector only processes one day's changes in the directory server.

The Sun Java System Directory Server Enterprise Edition 6.0 Administration Guide contains instructions on how to enable trimming of the retro changelog. Essentially, the nsslapd-changelogmaxage attribute in the cn=Retro Changelog Plugin, cn=plugins,cn=config entry is modified and Directory Server is restarted. Setting the value of this attribute to 1d trims entries that are older than one day. The value can be modified from the command line by using ldapmodify or by directly editing the Directory Server's dse.ldif file while Directory Server is not running.

Active Directory Connector

To allow Active Directory to detect changes that occurred while it was stopped, the Active Directory Connector persistently stores the usnChanged value of the last change that it processed. When Active Directory begins processing changes again, it only needs to examine entries with an usnChanged value larger than the previous one that it has processed. Like the Directory Server Connector, the Active Directory Connector first stores this usnChanged high-water mark when synchronization is started for the first time. Therefore, as part of preparing the failover installation, synchronization is started to allow the Active Directory Connector to check-point its state.

When failing over and synchronization is restarted for the failover installation, the Active Directory Connector will process all entries that were modified since it was last started. Even though this will be thousands of entries, the processing will be fast because the Active Directory object cache database is up-to-date with most of the changes. Only the changes made since the last time that the idsync resync command was run will be processed.

Initializing the Connector State

To initialize the connector state for the failover configuration, synchronization must be started. Before synchronization can be started, the Identity Synchronization for Windows Plug-in must be enabled on both master3-eu.gt.com and master4-eu.gt.com to point to the failover configuration. After the plug-in has been enabled and the directory servers have been restarted, synchronization can be started. Verify that both connectors have entered the SYNCING state, by using the console or the idsync printstat command:

bash-2.05# ./idsync printstat -w <password omitted\> -q <password omitted\>
Exploring status of connectors, please wait
Connector ID: CNN100
     Type: Sun Java(TM) System Directory
     Manages:  dc=gt,dc=com (ldaps://master3-eu.gt.com:636)
    (ldaps://master4-eu.gt.com:636)
     State: SYNCING
     Installed on:  connectors-eu.gt.com
     Plugin SUBC100 is installed on ldaps://master3-eu.gt.com:636
     Plugin SUBC101 is installed on ldaps://master4-eu.gt.com:636

Connector ID: CNN101
     Type: Active Directory
     Manages: gt.com (ldaps://ad1-us.gt.com:636) (ldaps://ad2-us.gt.com:636)
    (ldaps://ad4-eu.gt.com:636) (ldaps://ad3-eu.gt.com:636)
     State: SYNCING
     Installed on:  connectors-eu.gt.com

Sun Java(TM) System Message Queue Status:  Started

Checking the System Manager status over the Sun Java(TM) System Message Queue.

System Manager Status:  Started
SUCCESS

After synchronization has started, modify a user password in both Active Directory and Directory Server, which will force the connectors to persist their state. To verify, do the following:

• Directory Server Connector. On this connector, check for the presence of the /var/opt/SUNWisw/persist/ADP100/accessor.state file. Check that the highestacknowledgedchangenumber value stored in the file is not -1. (To determine the appropriate ADP subdirectory of persist, find the connector ID using the console or idsync printstat, then replace CNN with ADP in the connector ID.)

• Active Directory Connector. On this connector, check that it actually propagated the change. An INFO message that includes the usnchanged value should appear in the central log, for example:

  				[05/Nov/2004:14:07:38.982 -0600] INFO    18  CNN101 connectors-eu  
  				"The agent is sending the following inbound action to MQ: 
  				Type: MODIFY SUL: GT_USERS {Data Attrs: } {Other Attrs: cn: Jane Test 
  				dn: CN=Jane Test,CN=Users,DC=gt,DC=com objectclass: top,person, 
  				organizationalPerson, user dspswuserlink: Rwyr9YEFk0WYxbFP5Nnrjg== pwdlastset: 
  				127441696561778218 samaccountname: 3aa00test100001 sn: test100001 usnchanged: 120831 
  				whenchanged: 20041105230736.0Z passwordchanged: TRUE}." 
  				(Action ID=CNN102-1000A5846CB-5, SN=2)

After you have verified that both connectors have check-pointed their state, stop synchronization for the failover installation. Then, reinstall the Directory Server Plug-ins on master3-eu.gt.com and master4-eu.gt.com to point to the primary configuration.

Failover Installation Maintenance

After the connectors' state has been persisted, little maintenance is done on the failover installation. idsync resync -u should be run periodically as described in Periodic idsync resync Operation for Failover Installation.

If any additional attributes are added to the list of synchronized attributes, a full idsync resync -u operation should be executed, not an incremental one. If this is not done, when the Active Directory Connector is started after a failover, it will incorrectly detect modifications to the added attribute because its object cache database did not store the previous value.

When to Fail over

Identity Synchronization for Windows is a background system that with one exception is not user-facing. Therefore, if it is temporarily unavailable, for example, due to routine hardware maintenance, most users will be unaffected. After the system is restored, Identity Synchronization for Windows will synchronize all changes that were made while it was unavailable.

The user-facing aspect is the on-demand password synchronization performed from the Directory Server Plug-in to Active Directory. If on-demand password synchronization fails, the user will not be able to log in to Directory Server. Therefore, Identity Synchronization for Windows provides more availability options for this situation. The Directory Server Plug-in can be configured to authenticate to any Active Directory domain controller. Thus, even if all but one Active Directory domain controller is down, on-demand password synchronization will still succeed.

The Directory Server Plug-ins receive their configuration from the Directory Server Connector over an encrypted channel. This configuration, which includes the location of the Active Directory domain controllers and credentials, is cached in memory by the plug-in. So even if the Directory Server Connector is unavailable, it will still be able to connect to Active Directory. However, if Directory Server is restarted, the plug-in's cached configuration is lost, and on-demand synchronization on that Directory Server will fail until the Directory Server Connector is available.

Depending on the size of the deployment, the failover procedure might take minutes to over an hour to perform. Therefore, the failover procedure should not be undertaken if the Identity Synchronization for Windows outage is expected to be short and temporary, for example, during the system restart of the system that contains the Identity Synchronization for Windows Core. Use the failover procedure only in situations where Identity Synchronization for Windows must be completely reinstalled or a complete idsync resync operation must be run on a large population.

Such situations might include the following:

• Any machine that runs the Identity Synchronization for Windows Core or Connector experiences a hardware failure.

• The configuration directory that stores the Identity Synchronization for Windows configuration is corrupt.

• The Active Directory domain controller that the Active Directory Connector communicates with experiences a hardware fault and must be rebuilt.

• The preferred Directory Server is corrupted and must be initialized from another preferred Directory Server.

Failing Over

The failover process, at a high-level, involves only these tasks:

• Stopping Synchronization at the Primary Installation

• Starting Synchronization at the Failover Installation

• Re-enabling the Directory Server Plug-Ins

• Changing the PDC FSMO Role Owner

• Monitoring the Logs

Stopping Synchronization at the Primary Installation

Before starting synchronization at the failover installation, stop synchronization at the primary installation to prevent unwanted interaction between the two systems. Depending on the reasons for failing over, this is accomplished in different ways. If the primary installation of Identity Synchronization for Windows is still operating properly, for example, failing over because a domain controller or Directory Server is down, stop synchronization using the console or idsync stopsync. Otherwise, stop the Identity Synchronization for Windows daemon (on Solaris) or service (on Windows) on each system where Identity Synchronization for Windows is still operational.

Starting Synchronization at the Failover Installation

After synchronization is stopped at the primary installation, it must be started at the failover installation by using the console or idsync startsync.

The Directory Server Connector will not enter the SYNCING state until the Directory Server Plug-ins are reinstalled on the preferred and secondary Directory Servers (master3-eu.gt.com and master4-eu.gt.com).

The Active Directory Connector will need to process every entry in Active Directory that has been modified since it was last started. It might take several minutes for the Active Directory Connector to begin propagating changes to Directory Server. Setting the log level to INFO before starting synchronization can reduce the impact of the Active Directory Connector having to catch up.

Re-enabling the Directory Server Plug-Ins

To complete the failover process, the Directory Server Plug-in is re-enabled on each Directory Server, which ensures the following:

• The plug-ins running on the preferred Directory Servers use the encryption key from the failover installation to encrypt password changes.

• All directory servers receive an updated on-demand password synchronization configuration.

• Logging done by the plug-ins is sent to the Central Logger of the failover installation.

The plug-ins must be re-enabled in this order:

1. Failover installation's preferred Directory Server.

2. Failover installation's secondary Directory Server.

3. All other preferred and secondary Directory Servers.

4. All preferred and secondary Directory Server replicas.

The order in which the Directory Server Plug-ins are enabled is important. If they are enabled in the wrong order, on-demand synchronization requests could loop between two preferred Directory Servers, tying up all Directory Server connections.

When re-enabling the plug-ins, make sure to specify the configuration directory of the failover installation, for example, config-eu.gt.com.

This re-enabling procedure can be automated by doing more work ahead of time:

1. Install the Directory Server Plug-ins for the failover configuration.

2. Export the plug-ins' configuration for each master from the cn=pswsync,cn=plugins,cn=config tree.

3. Re-enable the Directory Server Plug-ins for the primary configuration.

To fail over:

1. Delete the cn=pswsync,cn=plugins,cn=config tree.

2. Add the failover installation entries by using ldapmodify.

3. Restart the directory server.

Changing the PDC FSMO Role Owner

This task is optional. If the Active Directory Connector in the failover installation is configured to communicate with a domain controller that does not have the PDC FSMO role, synchronization from Active Directory will be delayed due to the Active Directory replication latency. To avoid this delay, the PDC FSMO role can be transferred to the domain controller that the connector is accessing.

Monitoring the Logs

After the failover process is complete, monitor the central error log of the failover installation for any unexpected warnings. Warnings similar to the following will likely appear:

[08/Nov/2004:07:58:24.803 -0600] WARNING 25  CNN100 connectors-eu  
"Unable to obtain password of user CN=Jane Test,OU=people,DC=gt,DC=com, 
because the password was encoded by a previous installation of 
Identity Synchronization for Windows Directory Server 
Plugin. The password of this user cannot be synchronized at this time. 
Update the password of this user again in Directory Server."

These warnings occur for each password update in the retro changelog that was made before the Directory Server Plug-in was reinstalled because the Primary Directory Server Plug-in was configured to use a different encryption key from the failover Directory Server Plug-in. Many of these password updates were likely synchronized by the primary installation before it went offline, but those that occurred after the primary installation went offline cannot be recovered. Users who are affected must change their password either in Active Directory or Directory Server to synchronize their passwords.

Failing Back to the Primary Installation

The procedure for failing back to the primary installation is identical.