Monitoring the HTTP Load Balancer Plug-in

Configuring Log Messages

The load balancer plug-in uses the web server’s logging mechanism to write log messages. The default log level on the Application Server is set to the default logging level on Sun Java System Web Server (INFO), Apache Web Server (WARN) and Microsoft IIS (INFO). The application server log levels, FINE, FINER, and FINEST, map to the DEBUG level on the web server.

These log messages are written to the web server log files, and are in the form of raw data that can be parsed using scripts, or imported into spreadsheets to calculate required metrics.

Types of Log Messages

The load balancer plug-in generates the following types of log messages:

Load Balancer Configurator Log Messages

These messages will be logged when you are using idempotent URLs and error page settings.

An output for idempotent URL pattern configuration contains the following information:

When the log level is set to FINE:

CONFxxxx: IdempotentUrlPattern configured <url-pattern> <no-of-retries> for web-module : <web-module>
When the log level is set to SEVERE:

CONFxxxx: Duplicate entry of Idempotent URL element <url-pattern> for webModule <web-module> in loadbalancer.xml."
When the log level is set to WARN:

CONFxxxx: Invalid IdempotentUrlPatternData <url-pattern> for web-module <web-module>

An output for error page URL configuration contains the following information (log level set to WARN):

CONFxxxx: Invalid error-url for web-module <web-module>

Request Dispatch and Runtime Log Messages

These log messages are generated while a request is being load balanced and dispatched.

An output for standard logging for each method start contains the following information (log level set to FINE):

ROUTxxxx: Executing Router method <method_name>
An output for router logs for each method start contains the following information (log level set to INFO):

ROUTxxxx: Successfully Selected another ServerInstance for idempotent request <Request-URL>
An output for runtime logs contains the following information (log level set to INFO):

RNTMxxxx: Retrying Idempotent <GET/POST/HEAD> Request <Request-URL>

Configurator Error Messages

These errors appear if there are configuration problems, for example, if the custom error page referenced is missing.

Log level set to INFO:

ROUTxxxx: Non Idempotent Request <Request-URL> cannot be retried

For example: ROUTxxxx: Non Idempotent Request http://sun.com/addToDB?x=11&abc=2 cannot be retried
Log level set to FINE:

RNTMxxxx: Invalid / Missing Custom error-url / page: <error-url> for web-module: <web-module>

For example: RNTMxxxx: Invalid / Missing Custom error-url / page: myerror1xyz for web-module: test

Enabling Load Balancer Logging

The load balancer plug-in logs the following information:

Request start/stop information for every request.
Failed-over request information when the request fails over from an unhealthy instance to a healthy instance.
List of unhealthy instances at the end of every health check cycle.

Note –

When load balancer logging is enabled, and if the web server logging level is set to DEBUG or to print verbose messages, the load balancer writes HTTP session IDs in the web server log files. Therefore, if the web server hosting the load balancer plug-in is in the DMZ, do not use the DEBUG or similar log level in production environments.

If you must use the DEBUG logging level, turn off load balancer logging by setting require-monitor-data property to false in loadbalancer.xml.

To Turn on Load Balancer Logging

Set the logging options in the web server. The procedure depends on the web server:
- With Sun Java System Web Server
  
  In the server’s Admin console, go to the Magnus Editor tab and set the Log Verbose option to On.
- For Apache Web Server, set the log level to DEBUG.
- For Microsoft IIS, set the log level to FINE in the sun-passthrough.properties file.

Set the load balancer configuration’s monitor option to true.

Use the asadmin create-http-lb-config command to set monitoring to true when you initially create the load balancer configuration, or use the asadmin set command to set it to true later. Monitoring is disabled by default.

Understanding Monitoring Messages

The format of the load balancer plug-in log messages is as follows.

The start of an HTTP request contains the following information:

RequestStart Sticky(New) <req-id> <time-stamp> <URL>

The timestamp value is the number of milliseconds from January 1, 1970. For example:

RequestStart New 123456 602983 http://austen.sun.com/Webapps-simple/servlet/Example1
The end of an HTTP request contains the RequestExit message, as follows:

RequestExit Sticky(New) <req-id> <time-stamp> <URL> <listener-id> <response-time> Failure-<reason for error>(incase of a failure)

For example:

RequestExit New 123456 603001 http://austen.sun.com/Webapps-simple/servlet/Example1 http://austen:2222 18

Note –
In the RequestExit message, response-time indicates the total request turnaround time in milliseconds, from the perspective of the load balancer plug-in.
The list of unhealthy instances, as follows:

UnhealthyInstances <cluster-id> <time-stamp> <listener-id>, <listener-id>...

For example:

UnhealthyInstances cluster1 701923 http://austen:2210, http://austen:3010
A list of failed-over requests, as follows:

FailedoverRequest <req-id> <time-stamp> <URL> <session-id> <failed-over-listener-id> <unhealthy-listener-id>

For example:

FailedoverRequest 239496 705623 http://austen.sun.com/Apps/servlet/SessionTest 16dfdac3c7e80a40 http://austen:4044 http://austen:4045