mod_oc4j: Failed to find a failover oc4j process for session request for destination oc4j_socket_recvfull timed out

mod_oc4j: Failed to find a failover oc4j process for session request for destination

oc4j_socket_recvfull timed out

Have encountered an issues with oacore crashing in R12 Oracle Applications frequently:

Error:-

[Mon Mar 28 16:45:06 2011] [warn] [client 10.1.2.165] oc4j_socket_recvfull timed out
[Mon Mar 28 16:45:06 2011] [error] [client 10.1.2.165] [ecid: 1301310604:10.2.0.181:18046:0:3,0] mod_oc4j: request to OC4J our12-apps2.bilt.com:21510 failed: Connect failed
[Mon Mar 28 16:45:28 2011] [warn] [client 10.1.2.51] oc4j_socket_recvfull timed out
[Mon Mar 28 16:45:28 2011] [error] [client 10.1.2.51] [ecid: 1301310626:10.2.0.181:18104:0:1,0] mod_oc4j: request to OC4J our12-apps1.bilt.com:21510 failed: Connect failed
[Mon Mar 28 16:45:56 2011] [warn] [client 10.1.2.165] oc4j_socket_recvfull timed out
[Mon Mar 28 16:45:56 2011] [error] [client 10.1.2.165] [ecid: 1301310654:10.2.0.181:17174:0:8,0] mod_oc4j: request to OC4J our12-apps2.bilt.com:21510 failed: Connect failed
[Mon Mar 28 16:45:58 2011] [warn] [client 10.1.2.165] oc4j_socket_recvfull timed out
[Mon Mar 28 16:45:58 2011] [error] [client 10.1.2.165] [ecid: 1301310656:10.2.0.181:27684:0:1200,0] mod_oc4j: request to OC4J our12-apps2.bilt.com:21510 failed: Connect failed
Desc:
-------

The error 'mod_oc4j Failed to find a failover oc4j process for session request for destination' indicates that a session request was received by the Process Manger (PM). The PM was unable to find a java process to route the session to which would be defined as a "island" or "jgroup" and it failed.

The error indicates for some reason the OC4J process has become non responsive to the point where the PM has determined it failed or is no longer available. This error usually occurs when OHS (mod_oc4j) is not able to communicate ( over AJP ) with OC4J.

There are multiple reasons why this error may have happened: Some common reasons are

1. Load: e.g. Too many clients connecting to the application :
 - The number of requests are high enough that the process cannot respond to new requests prior to reaching the timeout period.

2. Performance: Heavily loaded JVM or Lack of enough memory for JVM
  - The OC4J oacore JVM processes in the middle tier are heavily loaded
  - High number of jdbc connections with database there by increasing the number of threads on the java process.


Solution :-

a) The number of jvms (oc4j in R12) is configured by the autoconfig variables s_oacore_nprocs, s_disco_nprocs, s_forms_servlet_nprocs and s_xmlsvcs_nprocs.

b) In 11i, those changes are made in the $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.conf file, while in R12 in the $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml

Heap Configuration:

1) For OACore, start with the following configuration:

For R12.0 and 11i:

       -Xmx512M -Xms256M -XX:MaxPermSize=128M
       -XX:NewRatio=2 -XX:+PrintGCTimeStamps

For R12.1 and higher:

      -Xmx1024M -Xms512M -XX:MaxPermSize=256M
      -XX:NewRatio=2 -XX:+PrintGCTimeStamps

     Also, ensure you add the following parameter to the DBC file:

     JDBC\:oracle.jdbc.maxCachedBufferSize=262144

   

Adjust the heap sizes accordingly to minimize garbage collection frequency and user pause times due to full garbage collections.
In JDK 1.6, the JVM detects that you have a server class machine (2 or more CPUs with 2GB or more memory), and will automatically enable Parallel Throughput Garbage Collector. The number of GC threads defaults to the number of CPUs on the machine. If you are running multiple JVMs on the same machine, or if your machine has more than 2 CPUs, to avoid the GC threads to be overly active you should reduce the number of GC threads by using:

      -XX:+UseParallelGC -XX:ParallelGCThreads=2

In most cases, using 2 GC threads should be sufficient. If you are using heap size of over 1GB, you can experiment with 4 GC threads and see if it gives you better performance.


2) If using Forms in Servlet mode, the following configuration for FormsGroup should be enough for most of the cases:

      -Xmx256M -Xms128M -XX:MaxPermSize=128M -XX:NewRatio=2

For More Information Please Ref:Guidelines to setup the JVM in Apps Ebusiness Suite 11i and R12 [ID 362851.1]