Part III Appendix

• Appendix A, Known Issues and Workarounds

• Appendix B, Error Messages

Appendix A Known Issues and Workarounds

The most common performance problems are identified by the following symptoms:

• Memory Grows or Leaks

• System Responds Too Slowly

• Server Hangs and Does Not Respond

• Server Crashes

Memory Grows or Leaks

• WS 6.1 bundled with JES 4 fail to start

WS 6.1 bundled with JES 4 fail to start

A memory leak may occur with securities libraries on Windows that were shipped with JES 4 for Windows. This causes the Web Server 6.1 (bundled with JES 4) to fail to start.

Solution:Install a standalone version of Weber Server 6.1 SP5 to force the Web Server to use its own bundled securities libraries from the webserver/bin/https/bin directory. Or you can use the JES 5 installer for installing Web Server 6.1SP5 or later.

System Responds Too Slowly

• Application Server where Access Manager is deployed throws "cannot create thread" error

• Directory Server 5.2 hangs and shows high CPU usage when deleting entries

• Throughput performance of AM is significantly slower when it is deployed on WebLogic or WebSphere Application Server.

Application Server where Access Manager is deployed throws "cannot create thread" error

You see an error saying "Cannot create thread" with the following stack trace:

"Access ManagerSessionPoller[9]" daemon prio=10 
tid=0x0985e2e0 nid=0x37 in Object.wait() [0x10519000..0x10519a38]
      at java.lang.Object.wait(Native Method)
      - waiting on <0x2ad92c18> (a java.util.ArrayList)
      at java.lang.Object.wait(Object.java:474)
      at com.iplanet.Access Manager.util.ThreadPool.getTask
          (ThreadPool.java:125)
      - locked <0x2ad92c18> (a java.util.ArrayList)
      at com.iplanet.Access Manager.util.ThreadPool$
          WorkerThread.run(ThreadPool.java:144)"

The problem is due to an insufficient amount of JVM heap size, or invalid Access Manager session threads are created out of control. This behavior is expected and not a deadlock at all.

Solution: To increase the JVM heap size, you can change the domain.xml manually or simply run Access Manager amtune-as8.

Directory Server 5.2 hangs and shows high CPU usage when deleting entries

You may see in the error log stating that "search is not indexed". The Directory Server referential integrity plug-in is automatically enabled by the Access Manager. But no indexes exist for Access Manager's attributes such as iplanet-Access Manager-static-group-dn and iplanet-Access Manager-modifiable-by. Access Manager does not configure the arguments of the plug-in, but uses the default arguments (update interval=0). Every deletion causes an immediate integrity check, which consumes a lot of system resources when the search is not indexed.

Solution: Be sure the referential integrity plug-in is enabled and configured, and that the attributes to be maintained are indexed. Be sure the referential integrity is configured to be executed synchronously or with a delay. A delay will remove the thread shortage per application.

If you observe that one or more of the deleted entries are repeatedly added or deleted, over time these entries may trigger non-indexed searches to the database. This issue is addressed in later versions of Directory Server. Upgrade to Directory Server 5.2 Patch 5 or Directory Server 6.0.

Throughput performance of AM is significantly slower when it is deployed on WebLogic or WebSphere Application Server.

Solution:First tune the JVM heap and GC (garbage collection) options for WebSphere Application Server. See Third-Party Web Containers for more information. Since JVM 1.4.2 or earlier is much slower than JVM 1.5.0 or later in throughput performance, make sure the JVM version used in the container is 1.4.2 or later when you use the CMS and NewParallelGC options. WebLogic 9.0 or later and WebSphere 6.0 or later use JDK 1.5.0 or later.

"OutOfMemoryError" When Access Manager is deployed in WebLogic or WebSphere Application Server

This occurs on Access Manager 7.0 or higher, even with sufficient JVM heap sizes.

Solution:Be sure that in addition to having sufficient JVM heap size, the CMS and NewParallel GC options are specified. Also be sure that -XX:-CMSParallelRemarkEnabled is included. See Third-Party Web Containers for more information.

Server Hangs and Does Not Respond

• Access Manager Server hangs during session failover

• Server hangs when processing request between the load balancer and the Access Manager server.

• Access Manager server hangs when Sun Java System Directory Server restarts

• Access Manager unable to recover after a crash or watch dog restart under heavy load

• jaxrpc getAttributes throws SSOException

• Sun Java System Web Server hangs while handling a large number of images files

• Access Manager Web Policy Agent hangs

• Access Manager server hangs when multiple clients point to one Access Manager server instance

• System hangs when Access Manager clientsdk.jar and Access Manager server are in the same JVM instance

Access Manager Server hangs during session failover

The problem occurs when both Access Manager server and JMQ (and even BDB) are installed in one machine. Both web server instances hang. A thread dump from the first web server instance shows all its threads are in socketRead operations.

waiting:

- java.net.SocketInputStream.socketRead0
(java.io.FileDescriptor, byte[], int, int, int) @bci=0, 
pc=0xf75e0274, methodOop=0xf33a7aa8 
(Compiled frAccess Managere; information may be imprecise)

A thread dump of the second web server instance shows the corresponding writePacketNoAck calls from jmsclient:

 
-com.sun.messaging.jmq.jmsclient.ProtocolHandler.writePacketNoAck
(com.sun.messaging.jmq.io.ReadWritePacket) @bci=7, line=235, 
pc=0xf79a56b4, methodOop=0xf3650320 (Compiled frAccess Managere; 
information may be imprecise)
-com.sun.messaging.jmq.jmsclient.ProtocolHandler.writeJMSMessage
(javax.jms.Message) @bci=565, line=1567, pc=0xf76bc190, 
methodOop=0xf36533c8 (Compiled frAccess Managere)
-com.sun.messaging.jmq.jmsclient.WriteChannel.sendWithFlowControl
(javax.jms.Message) @bci=10, line=123, pc=0xf7825278, 
methodOop=0xf3689e48 (Compiled frAccess Managere)
-com.sun.messaging.jmq.jmsclient.TopicPublisherImpl.publish
(javax.jms.Message) @bci=2, line=73, pc=0xf782ece0, 
methodOop=0xf36b2400 (Compiled frAccess Managere)
-com.iplanet.dpro.session.jmqdb.JMQSessionRepository.save
(com.iplanet.dpro.session.service.InternalSession) @bci=92, 
line=346, pc=0xf7775008, methodOop=0xf3604770 (Compiled frAccess Managere)
-com.iplanet.dpro.session.service.SessionService.saveForFailover
(com.iplanet.dpro.session.service.InternalSession) @bci=26, 
line=2485, pc=0xf7005c34, methodOop=0xf35da5e8 (Interpreted frAccess Managere)
- com.iplanet.dpro.session.service.InternalSession.updateForFailover() 
@bci=46, line=969, pc=0xf74a7c48, methodOop=0xf36c0420 
(Compiled frAccess Managere)

Under a heavy load, the Access Manager server web container process will use most of the machine's CPU resources. Then JMQ and/or BDB with Access Manager sessiondb will not have sufficient CPU resources to process incoming requests. The first Access Manager server instance's threads carrying requests cannot write to the second Access Manager server instance with JMQ in the back because of the lack of CPU resources. Also the first Access Manager server instance will have its threads built up because of the backlog on the second instance due to the lack of processing on the part of JMQ and/or BDB for updating the session table.

Solution: Install JMQ and BDB on their own boxes, separate from Access Manager server machine.

Server hangs when processing request between the load balancer and the Access Manager server.

The problem occurs when using two Access Manager 7.0 SP5 servers with a load balancer in front. You may see a stack trace such as this:

" at sun.net.www.protocol.http.HttpURLConnection.getInputStream
(HttpURLConnection.java:961)
   - locked <0xf0898b88> (a sun.net.www.protocol.http.HttpURLConnection)
*     *at com.iplanet.services.comm.client.PLLClient.send(PLLClient.java:196)
   at com.iplanet.services.comm.client.PLLClient.send(PLLClient.java:115)"

Solution: The problem might be a result of having the server instances separated by a firewall. If this is your environment, move the server instances behind the same firewall.

The problem could be due to a misconfiguration in the Platform Server List. The stack trace shown above occurs when the Platform Server List is missing its associated site ID's and server instances are denoted by virtual host names. Here is an example of a misconfigured Platform Server List:

site list : http://hostname:80|11
server list : http://hostname1:80|01
           		http://hostname2:80|02

Configure your Platform Server List to include the server identifiers. In the following example, 11 is the server identifier.

site list : http://hostname:80|11
server list : http://hostname1:80|01|11
           		http://hostname2:80|02|12

Access Manager server hangs when Sun Java System Directory Server restarts

Connections between Access Manager server and Directory Server seem to close unexpectedly due to unsynchronized access to a shared variable. This is a known problem that is fixed in LDAP JDK 4.19

Solution: Upgrade to LDAP JDK 4.19. Download Patch 119725–04–1 from the following URL: http://sunsolve.sun.com/search/document.do?assetkey=1-21-119725-04-1 The patch will work for Solaris 9 and 10. The same patch is used for both SPARC and x86 platforms.

Access Manager unable to recover after a crash or watch dog restart under heavy load

This problem occurs when using an LDAP v3 plug-in because the plug-in is being initialized more than once.

Solution: This known problem was fixed in both Access Manager 7.0 Patch 6 and Access Manager 7.1 Patch 1. Upgrade to one of these Access Manager versions.

jaxrpc getAttributes throws SSOException

A 403 error occurs. The problem occurs when session expiration notifications come between the SSO validation call and the call to fetch profile attributes on the agent side. The SSO validation is successful, but the profile attribute fetch fails with an SSOException from the server. This is the expected behavior. However, the SOAP client is not processing this exception properly, and is re-constructing. The client side calls wrap up this exception with IdRepoException. As a result the agent is not notified about the SSOException.

Solution:A fix was made in amclientsdk. This known problem was fixed in both Access Manager 7.0 Patch 6 and Access Manager 7.1 Patch 1. Upgrade to one of these Access Manager versions.

Sun Java System Web Server hangs while handling a large number of images files

The web container that hosts Access Manager hangs while handling a large number of image files. The following errors are display in the Web Server errors.log

IO error:
           "java.io.IOException: WEB8004: Error flushing the output stream
            java.io.IOException: WEB8001: Write failed "

These entries indicate that Access Manager server was not involved in the hang of the web server but instead the root cause was due to the an IO error.

Access Manager Web Policy Agent hangs

When a web policy agent hangs, it is usually due to misconfiguration of the web container where the agent is installed, or misconfiguration of the Access Manager web container on another host system. An Access Manager server may be running out of some resources due to, for example, a runaway number of invalid sessions.

Solution: Set the Web Policy Agent debug logging level to the finest level, all:5. Examine the logs to determine the exact cause.

Access Manager server hangs when multiple clients point to one Access Manager server instance

When multiple clients point to an Access Manager server instance, the session polling mechanism used prior to Access Manager 7.1, can cause the Access Manager server to hang. The older polling mechanism was based on caching time.

Solution: Upgrade to Access Manager 7.1. In the 7.1 version, the session polling mode is configurable. You can use on either caching or idle time mode. By default it is based on the idle time.

System hangs when Access Manager clientsdk.jar and Access Manager server are in the same JVM instance

The problem occurs when both a client application with Access Manager clientsdk.jar and Access Manager server are in the Access Manager JVM instance. When an JavaEE application tries to access IdRepo attributes for any identity, then Access Manager server can hang. The problem is due to the unnecessary synchronization block in SMS ServiceConfigImpl, OrganizationConfigManagerImpl and ServiceConfigManagerImpl in Access Manager SDK. This known issue was address in Access Manager 7.1.

Solution: Upgrade to Access Manager 7.1

Server Crashes

• Access Manager web container crashes with "StackOverFlowError" errors

• Apache Web Agent 2.2 on Linux crashes

• Access Manager crashes in SSL mode

• Customized JSP page causes Web Server to crash

• Application Server or Web Server crashes under a heavy load

Access Manager web container crashes with "StackOverFlowError" errors

The problem is known to occur when Access Manager 7.0 is deployed a web server, and the "-Xss128k" JVM option is used with 64-bit JVM on the web container. The problem can occur with any java application. For 64-bit JVM's, the minimum per thread stack size should be 256k, -Xss256k, or even 512k since 64-bit VM's default per thread stack size is 1 mb.

Solution: For 64-bit JVMs, the minimum per thread stack size should be 512k since the 64-bit JVM default per thread stack size is 1 mb. The 64-bit JVM support was introduced starting with Web Server 6.1 SP5. Application Server 8.1 and 8.2 do not support 64-bit JVM, but Application Server 9.1 will. Access Manager and its amtune scripts support 64-bit JVM starting with AM 7.0 Patch 5. For more information, see http://java.sun.com/docs/hotspot/threads/threads.html

Apache Web Agent 2.2 on Linux crashes

Apache Web Server crashes when Web Agent is deployed.

Solution: There is no solution at this time. Running the Apache Server in multi-process mode (MPM) of compilation and runtime modes are not supported by our Apache Agent in Access Manager.

Access Manager crashes in SSL mode

The problem occurs When Access Manager server is configured in SSL mode and there is outbound SSL traffic from Access Manager server to another Access Manager server or Directory Server. In some rare situations where SSL socket calls don't get closed and queue up, the Access Manager server can crash if it is configured in the default NSS/JSS mode of SSL.

Solution: Upgrade to JSS 4.2.5.

Customized JSP page causes Web Server to crash

Web Server crashes when a customized JSP page for Access Manager is deployed in Sun Java System Web Server. The problem occurs when using a Web Server version prior to 6.1 SP8. The problem is due to a known problem in Sun Web Server that is unrelated to Access Manager server. If the JSP has calls to HttpServlet.getScheme or HttpServlet.service, the Web Server can crash.

Solution: Upgrade to Web Server 6.1 SP8.

Application Server or Web Server crashes under a heavy load

The problem is known to occur when using Sun Fire T1000/T2000 hardware (cool threads, Niagara boxes) to deploy Access Manager server with Sun Java System Application Server or Sun Java System Web Server.

Solution: Set the appropriate memory library the Application Server asenv.conf file or in the Web Server start script. For Application Server, in the asenv.conf file, replace LD_PRELOAD=/usr/lib/libmtmalloc.so with LD_PRELOAD=/usr/lib/libumem.so. For more information, see http://www.sun.com/servers/coolthreads/tnb/applications_sunone.jsp.

For Web Server in the start script, replace LIBMTMALLOC=/usr/lib/libmtmalloc.so with LIBMTMALLOC=/usr/lib/libumem.so..

Use the latest JDK 1.5 version, at least 1.5.0_08 or later when Sun Fire T1000/T2000 boxes are used for Access Manager server deployments.

Appendix B Error Messages

The following are Access Manager error messages you may encounter in log files for Access Manager, Web Server, Directory Server, Portal Server, or the Policy Agent host.

Error Log for the J2EE Policy Agent Application Server

thread dump from "kill -3" command shows hundreds of waiting threads like: service-j2ee" daemon prio=10 tid=0x00d59180 nid=0x11e waiting for monitor entry//at com.sun.identity.jaxrpc.SOAPClient.encodeMessage//- waiting to lock <0x787e3ad0> (a com.sun.identity.jaxrpc.SOAPClient)

Description:

Error message is found in the J2EE Policy Agent web container log.

Cause:

This issue is caused by an unnecessary synchronization in the SOAP client Java class in amclientsdk for encode and send methods. During a load test of URL policy mode with J2EE policy agents, hundreds of threads can be waiting to lock on the com.sun.identity.jaxrpc.SOAPClient.send method.

Solution:

Two related bugs (CR 6302120 and CR 6517760) were fixed in Access Manager 7.0 Patch 5 and Access Manager 7.1 Patch 1.

Web Policy Agent amAgent.log File

Error 30284:bfc093a0 all: Connection::read(): NSPR Error while reading data:-5961

Description:

The Access Manager server is busy responding to all the previous requests from a web policy agent, and cannot respond to this particular request. Then the socket timeout happens on the web policy agent side, and the user will see this error message in the web policy agent amAgent.log.

Cause:

The agent has timed out waiting for a response from the Access Mnager server.

Solution:

Be sure the Access Manager server is properly tuned with values recommended in the amtune script. Also be sure that the web agent HTTP request parameters are properly tuned.

Web Policy Agent amAgent.log File

Error 19516:9088eb0 AM_SSO_SERVICE:SSOTokenService::getSessionInfo(): Error 18 for sso token ID Error 21907:9088eb0 PolicyEngine: am_policy_evaluate: InternalException in Service::initialize() with error message: Naming query failed during service creation. and code:21"

Description:

These errors occur during stress tests of Web Agent 2.2 for Apache Server 2.0.59 on RedHat Linux 3.0.

Cause:

These errors mean that the SSO token has expired on the server side, but the agent is still sending the expired SSO token. In normal cases, if the web policy agent sees this error, it will redirect to the Access Manager login page. The Access Manager server becomes overwhelmed from all the incoming requests from the web policy agent. Other errors may occur:

Error 30284:bfc093a0 all: Connection::read(): NSPR Error while reading data:-5961 Error 30154:bfc093a0 all: Connection::read(): NSPR Error while reading data:-5961 Error 30054:bfc093a0 all: Connection::read(): NSPR Error while reading data:"

These errors occur because the web policy agent has timed out waiting for a response from the Access Manager server. During this load test the Access Manager server was so busy responding to all the previous requests, it failed to respond to this particular request. Then the socket timeout happens on the agent side and the user will see this error message.

Solution:

Be sure the Access Manager server is properly tuned with amtune script recommended values. Also be sure that the web agent HTTP request parameters are properly tuned.

Access Manager amclientSDK File or Error Log for the J2EE Policy Agent

ERROR: Send Polling Error:com.iplanet.am.util.ThreadPoolException: amSessionPoller thread pookdkdkdkl's task queue is full.

Description:

These errors can occur after you deploy the Distributed Authentication UI web application, J2EE agents, or in any situation where you deploy the Access Manager client SDK on a client machine. There errors are seen in the J2EE Agent web container log or in the amclientSDK container log.

Cause:

The client SDK polling threadpool size and threshhold are not sufficient for the number of incoming sessions.

Solution:

If you have many concurrent sessions, add the following properties and values in either the AMConfig.properties file or the AMAgents.properties file:

• com.sun.identity.session.polling.threadpool.size=10

• com.sun.identity.session.polling.threadpool.threshold=10000

Access Manager amSession.log File

ERROR: Sending Notification Error:com.iplanet.am.util.ThreadPoolException: amSession thread pool's task queue is full.

Description:

These errors can occur when the number of incoming Access Manager sessions is more than the notification threadpool size and threshold can handle.

Cause:

The AMConfig.propertiesdefault values for com.sun.identity.session.notification.threadpool.size and com.sun.identity.session.notification.threadpool.threshold are too low.

Solution:

Increase the value for notification.threadpool.size to be three times the number of CPUs, or cores in case of Niagara boxes, where the Access Manager server is installed. Increase the value for notification.threadpool.threshold or the queue to be 30% of maxSessions in the AMConfig.properties file . If the same error still occurs, then an agent may not processing incoming requests efficiently, or some other bottleneck exists on the client side or on the Access Manager web container.

Access Manager amSession.log File

ERROR: Individual notification to http://mycompany.com:7001/agentapp/notificationcom.iplanet.services. comm.server.SendNotificationException: Server returned HTTP response code: 503 for URL: http://yourcompany.com:7001/agentapp/notification

Description:

These errors occur during a high load scenario when a bottleneck in the notification queue exists on the port between the Access Manager server and its policy agent machines.

Cause:

The notification attempts to the agent were not successful.

Solution:

There is no solution for this now. But starting with Federated Access Manager 8.0, notification traffic will use JMQ asynchronous publish and subscribe mechanisms with different ports, which will eliminate this kind of bottleneck.

Access Manager amComm.log File

ERROR: Cannot send notification to http://mycompany.com:80/amagent/UpdateAgentCacheServlet?shortcircuit= false java.io.IOException:Server returned HTTP response code: 503 for URL: http://yourcompany.com:80/amagent/UpdateAgentCacheServlet?shortcircuit=false

Description:

These errors occur during a high load scenario when a bottleneck in the notification queue exists on the port between the Access Manager server and its policy agent machines.

Cause:

The notification attempts to the agent were not successful.

Solution:

There is no solution for this now. But starting with Federated Access Manager 8.0, notification traffic will use JMQ asynchronous publish and subscribe mechanisms with different ports, which will eliminate this kind of bottleneck.

Policy Agent amAgent.log File

Info PolicyAgent: am_web_result_attr_map_set(): No profile or session or response attributes to be set as headers or cookies Debug all: Log::pSetLevelsFromString(): setting log level for module 0 to 4, old level 1.

Description:

This error means the session or response attribute is missing from URL string. The Web Server crashes.

Cause:

Insufficient size of the maximum length on the URL (query string length) that can be passed to web policy agent.

Solution:

Upgrade to Web Policy Agent 2.2-HP8.

SAML2 Debug Log

This log is stored in the following location:

/var/opt/SUNWam/fm/federation/debug/fmSAML2

ERROR: Unable to send SOAPMessage to IDP com.sun.xml.messaging.saaj.SOAPExceptionImpl: java.security.PrivilegedActionException: com.sun.xml.messaging.saaj.SOAPExceptionImpl: Unable to internalize message at com.sun.xml.messaging.saaj.client.p2p.HttpSOAPConnection.call at com.sun.identity.saml2.common.SAML2Utils.sendSOAPMessage at com.sun.identity.saml2.profile.LogoutUtil.doSLOBySOAP

Description:

In a high availability and high load scenario, for example when more than one Access Manager server or Federation Manager are behind a load balancer, the SOAP Global logout fails if it redirects to the wrong server.

Cause:

The signature string is not forwarded when redirected to the internal server instance.

Solution:

This bug was fixed in SAML v2 Patch 3. The fix delays the signature validation until the Access Manager or Federation Manager server finds the session in the local server. This way there is little processing involved before the signature verification is done. Update to SAML v2 Patch 3.

SAML2 Debug Log

This log is stored in the following location:

/var/opt/SUNWam/fm/federation/debug/fmSAML2

ERROR: Unable to get infoKeyString from SSOToken. ERROR: Error sending Logout Request com.sun.identity.saml2.common.SAML2Exception: Error retrieving NameIdInfoKey from SSOToken. at com.sun.identity.saml2.profile.SPSingleLogout.initiateLogoutRequest at _jsps._saml2._jsp._spSingleLogoutInit_jsp._jspService

Description:

This error will sometimes occur during a high load scenario even with SAML v2 Patch3. The NameIDInfoKey information is stored in session properties, but sometimes during a high load scenario, the information cannot be retrieved.

Cause:

The session properties do not get refreshed immediately.

Solution:

Refresh the session properties before reading the NameIDInfoKey.

Error Log for Web Server or Application Server

SEC_ERROR_NO_MEMORY: Out of memory

Description:

Web Server or Application Server process crashes. This crash will occur only if SSL is enabled for the Web Server or Application Server.

Cause:

A bug exists in NSPR 4.5. The NSPR threads created in linux use 10240kb as the stack size regardless of the stack size specified during thread creation. The default is 10240 kb per thread stack on the Red Hat Linux platform.

Solution:

Upgrade the NSPR version to 4.6.

Error Log for Web Server or Application Server

Cannot create thread. amSessionPoller [9] daemon prio=10 tid=0x0985e2e0 nid=0x37 in Object.wait () [0x10519000..0x10519a38 at java.lang.Object.wait (Native Method)

Description:

Web Server or Application Server processes cannot create any more threads for Access Manager sessions.

Cause:

Insufficient JVM heap size or invalid Access Manager session threads are created out of control. This behavior is expected.

Solution:

To increase the JVM heap size, change the domain.xml manually or run the Access Manager amtune-as8 script.

Error Log for Web Server or Application Server

java.IOException:Not enough space at java.lang.UNIXProcess.forkAndExec(Native Method) at java.lang.UNIXProcess.forkAndExec at java.lang.UNIXProcess

Description:

The JVM cannot launch itself while trying to fork a process through the system.

Cause:

Either there are not enough file descriptors, or there is not enough swap space.

Solution:

Do one of the following:

• Increase the number of system file descriptors, then reboot the machine. To increase the number of file descriptors, you can run the amtune-os script, manually set them by running the command ulimit -n number_of_file_descriptors.

• Increase the swap space by killing unnecessary processes.

• Add more swap space using swap command.

Error Log for Web Server or Application Server

Exception in thread "service-j2ee" java.lang.OutOfMemoryError: requested 53515 bytes for jbyte in /BUILD_AREA/jdk1.5.0_10/hotspot/src/share/vm/prims/jni.cpp. Out of swap space?

Description:

The native heap allocation failed and the native heap may be close to exhaustion.

Cause:

A native code leak, for example the C or C++ code, continuously requires memory without releasing it to the operating system. There could be indirect causes like an insufficient amount of swap space or another process that is consuming all memory or leaking it.

Solution:

For further diagnosis of a native code memory leak, see the Java 5.0 Troubleshooting and Disgnostic Guide at http://java.sun.com/j2se/1.5/pdf/jdk50_ts_guide.pdf. In the section “Diagnosing Leaks in Native Code.” See the information about tools for different operating systems. The tools include mdb and dbx (runtime trace) for Solaris 9 U3 or later, mtrace , libnjamd for Linux, and windb or userdump for Windows."

Error Log for Web Server or Application Server

Exception in thread "main" java.lang.OutOfMemoryError: <reason> <stack trace>(Native method)

Description:

A native method has encountered a memory allocation failure. The difference between this and the previous error message is that the allocation failure here is detected in a JNI or native method rather than VM code.

Cause:

A native code leak, for example the C or C++ code, continuously requires memory without releasing it to the operating system. There could be indirect causes like an insufficient amount of swap space or another process that is consuming all memory or leaking it.

Solution:

For further diagnosis of a native code memory leak, see the Java 5.0 Trouble-Shotting and Disgnostic Guide section “Diagnosing Leaks in Native Code.”

Error Log for Web Server or Application Server

Exception in thread ?main? java.lang.OutOfMemoryError: Java heap space

Description:

An object could not be allocated in the Java heap.

Cause:

The cause is a simple configuration issue. The maximum heap size, noted by -mx options is not sufficient for the load. Or the application may be holding references to objects which cannot be garbage collected. This is a Java equivalent of a memory leak. If the finalize method is used so much that is that the finalizer daemon thread cannot keep up with the finalization queue, then this error can occur when the heap becomes full.

Solution:

Maximum JVM option increase or coding changes.

Error Log for Web Server or Application Server

Exception in thread ?main? java.lang.OutOfMemoryError: PermGen space

Description:

This error occurs when there are a large number of class, method or String objects.

Cause:

The permanent generation is full. The permanent generation is the area of the heap where class and method objects are stored and java.lang.String objects are interned.

Solution:

The JVM option for Perm size may need to be increased.

Error Log for Web Server or Application Server

Exception in thread "main" java.lang.StackOverflowError at java.lang.String.indexOf

Description:

The JVM (java) stack size is not sufficient

Cause:

There can be many types of StackOverflowError errors including a wrong server instance name in the platform list on the Access Manager console, or any one of numerous Java coding issues. But here the only type of StackOverflowError that will be addressed is the one that can occur when you use 64-bit JVM with the -Xss128k option.

Solution:

For 64-bit JVM's, the minimum per thread stack size should be at least 256k, -Xss256k, or even 512k since 64-bit VM's default per thread stack size is 1 mb. 64-bit JVM support was introduced starting with Web Server 6.1 SP5 or later, including Web Server 7.0. Application Server 8.1 and 8.2 do not support 64-bit JVM, but Application Server 9.1 will. Access Manager and its amtune scripts support64-bit JVM, starting with AM 7.0 Patch 5 and 7.1. For more information, see http://java.sun.com/docs/hotspot/threads/threads.html.