JNDI Connection Pool stops sending traffic while establishing connections

[Jackson, Gerald]

The JNDI connection pool stops sending traffic while establishing a connection.  This occurs both while the connection pool is initially being created (and the minimum number of connections are being established), and when the pool is active but a connection terminates / new connection is therefore established.



The JNDI pool should continue to send traffic while establishing a connection.



We are seeing intermittent long connection establishement time (3 seconds), and the JNDI connection pool stops sending to our LDAP cluster during that delay.


(Our particular delay is due to SNAT behavior at the load balancer in front of our LDAP pool.  Occasionally a client port will appear, to an LDAP node, to conflict with a port that it believes to be in TCP TIME_WAIT state.  This ends up resulting in a RESET, followed by a 3 second delay in client-side TCP, then successful establishment of the TCP connection).

The port behavior that we see will only occur when there's an F5 in the way, and multiple server nodes making directory requests.  For example:
ESecMgr1: IP 1.1.1.1
ESecMgr2: IP 2.2.2.2
F5: IP 3.3.3.3

·         ESecMgr1: connects to the directory from 1.1.1.1:4000.  After F5 SNAT, directory sees this as coming from 3.3.3.3:4000 (because the F5 prefers to keep the original source port).

·         That connection completes, but is remembered at the directory node in TIME_WAIT. (Our F5 setup uses fastl4, so it just passes packets through, and does not impose its own TIME_WAIT).

·         ESecMgr2: connects to the directory from 2.2.2.2:4000.  The directory sees this as coming from 3.3.3.3:4000, so it ACKs rather than SYN/ACKing, triggering the RST and delay that we see.


Version: jdk1.6.0_38
Type: Production
Operating System: Red Hat Enterprise Linux Server release 5.9 (Tikanga)

Issue Type: Defect
Summary: JNDI Connection Pool stops sending traffic while establishing connections
Severity: Severity 2
Severity Notes:
While this is being categorized as a Sev 2, it's causing a issues to our clients ~.005% (or great) of the time.
Until we resolve this issue, we are unable to bring more clients on to our application.

Detailed Description:
jndi package - javax.naming
Connection Pool Class - javax.naming.directory.InitialDirContext

Connection Pool Settings
------------------------------------
<prefPoolSize>30</prefPoolSize>
<initPoolSize>30</initPoolSize>
<maxPoolSize>30</maxPoolSize>
<poolConnectTimeout>300000</poolConnectTimeout>


Architecture
------------------
ESecMgr App àF5 à Directories

ESecMgr App

-          uses JNDI Connection Pool

-          12 instances

F5

-          The VIP is configured to use the fastl4 profile, using address translation with a preference for retaining the original source port.

-          Because this is fastl4, there's no time wait setting on the F5 (connection establishment/termination just flows through the F5 basically unmodified).

directories behind F5

-          6 instances




Thanks in advance.

Gerald R. Jackson
Security Systems | Enterprise Infrastructure | Sabre Holdings
office: 682.605.2954 | cell: 817.938.5508

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/jdk6-dev/attachments/20131022/867886f9/attachment.html