SMP JNI issue, UseMembar workaround resolves it

[Scott Valentine]

We ran into an issue where our application would consistently crash with a
segmentation violation after roughly 15 minutes to 90 minutes of runtime.
It's not exactly a bug, but I thought it would be helpful to post the
information here for other folks, and to hopefully support the great work
of OpenJDK developers down the road.

The quick details are that we consistently die without much error detail
(just a simple segmentation violation printout) when our code enters JNI,
does some stuff, and then calls back into the VM. The JNI_ENTRY fails when
calling transition_from_native.

The client application is running on an Asus Aspire-One netbook (Atom
N270, dual core @800MHz) with OpenJDK-1.6.0-20-1.9.7. A gdb stack trace
and jstack dump is attached for details on what is happening. More details
on the system structure are included below for those interested, but
basically it is a moderately threaded, intensively JNI application running
under the Equinox OSGi runtime.

It was a little tough to debug, as the clients are remote and I have to go
through multiple ssh back-doors. We initially suspected our JNI
middleware, but after getting the necessary debugging symbols, tools, and
builds in place, we found that it was always crashing on the
write_memory_serialize_page call when attempting JNI_ENTRY after spending
some time in the native code. It never even got to the point of reference
values like the VM env, jobject, etc. Anyhow, the source for the
transition_from_native call led us to try the -X:+UseMembar option which
seems to have resolved the issue.

Anyhow, I hope the trace info is helpful, and please let me know if I can
provide more info. I can't spare a ton of cycles, but I would be happy to
contribute as time permits.

Here are the application details:

As mentioned previously, the application is running in the Equinox OSGi
framework, and it relies heavily on two JNI libraries: the RXTX library
(2.1-7r2), and a middleware called opensplice DDS (5.4.1). Opensplice is a
shared memory model runtime that runs as three seperate processes, and has
a JNI interface into the framework. The application has two serial devices
(two RXTX threads), and we have a thread for each (two more threads) that
does blocking reads on those ports. These threads put data into a
BlockingQueue, which has another thread that takes data from the queue and
processes it (two more threads). These threads process the data, make JNI
calls into the DDS middleware (this is where the failures have, at least
so far, always occurred), and put some information into another Blocking
Queue. There are two other application threads (total of eight now). The
first periodically writes to one of the serial port. The other thread
handles the second blocking Queue and also makes JNI calls into the DDS
middleware. Overall, there are three threads calling into that middleware
independantly.

I think there are something like 20 threads total, but three are the JVM
threads, and 7 or so are related to Equinox and our launcher that don't
really do anything unless the system is starting or stopping or doing
something in the OSGi world.

Thanks, and again, I hope this info can be helpfult to others.

Scott Valentine

Concentris Systems LLC
Manoa Innovation Center, Suite #238
2800 Woodlawn Drive
Honolulu, HI  96822

http://www.Concentris-Systems.com

(808) 988-6100

-------------- next part --------------
A non-text attachment was scrubbed...
Name: jstack_trace.log
Type: application/octet-stream
Size: 11247 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-dev/attachments/20110610/38e69950/attachment-0003.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jdb.log
Type: application/octet-stream
Size: 3876 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-dev/attachments/20110610/38e69950/attachment-0004.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gdb.log
Type: application/octet-stream
Size: 5044 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-dev/attachments/20110610/38e69950/attachment-0005.obj