Analysis on JDK-8022321 java/lang/ref/OOMEInReferenceHandler.java fails intermittently

[srikalyan]

Hi all,  I have been working on the bug JDK-8022321 
<https://bugs.openjdk.java.net/browse/JDK-8022321> , this is a sporadic 
failure and the webrev is available here 
http://cr.openjdk.java.net/~srikchan/Regression/JDK-8022321_OOMEInReferenceHandler-webrev/
*
**"Root Cause:Still not known"*
2 places where there is a possibility for OOME
1) Cleaner.clean()
2) ReferenceQueue.enqueue()

1)  The cleanup code in turn has 2 places where there is potential for 
throwing OOME,
     a) thunk Thread which is run from clean() method. This Runnable is 
passed to Cleaner and appears in the following classes
         java/nio/DirectByteBuffer.java
         sun/misc/Perf.java
         sun/nio/fs/NativeBuffer.java
         sun/nio/ch/IOVecWrapper.java
         sun/misc/Cleaner/ExitOnThrow.java
However none of the above overridden implementations ever create an 
object in the clean() code.
     b) new PrivilegedAction created in try catch Exception block of 
clean() method but for this object to be created and to be held 
responsible for OOME an Exception(other than OOME) has to be thrown.

2) No new heap objects are created in the enqueue method nor anywhere in 
the deep call stack (VM.addFinalRefCount() etc) so this cannot be a 
potential cause.

*Experimental change to java.lang.Reference.java* :
- Put one more guard (try catch with OOME block) in the Reference 
Handler Thread which may give the Reference Handler a chance to cleanup. 
This is fixing the test failure (several 1000 runs with 0 failures)
- Without the above change the test fails atleast 3-5 times for every 
1000 run.

*PS*: The code change is to a very critical part of JDK and i am fully 
not aware of the consequences of the change, hence seeking expert help 
here. Appreciate your time and inputs towards this.

-- 
--
Thanks
kalyan
Ph: (408)-585-8040