RFR: JDK-8066859 java/lang/ref/OOMEInReferenceHandler.java failed with java.lang.Exception: Reference Handler thread died

[Peter Levart]

Hi,

I might have a closure for this long beard issue [1] and I'd like to 
propose a change [2].

It all started with [3] discussed here [4] which resulted in the fix for 
one source of OutOfMemoryError(s) terminating ReferenceHandler thread. 
With this fix, new test was added which started failing intermittently 
indicating there is another source of OutOfMemoryError(s) being thrown 
in ReferenceHandler. At that time, the investigation [6] indicated that 
the source is "r instanceof Cleaner" check and so another change was 
pushed [5]. The issue JDK-8022321 is internal - I don't have a link. 
That change was 2-fold: it added code to preload and pre-initialize 
Cleaner and InterruptedException classes in ReferenceHandler.<clinit> 
and, cowardly, surrounded "r instanceof Cleaner" check with OOME 
exception handler. I'm pretty certain that pre-loading of the classes 
would have been enough. At least it would have not sprung the confusion 
expressed here [7].
But even that was not enough to prevent intermittent failures. I 
realized that the test I devised still fails because of Cleaner.clean() 
throwing OOME. Why this is possible is discussed in the issue [1]. I 
even modified the test to reproduce such failure with 100% chance.

So here I propose two things in [2] :
- roll-back of the confusing "r instanceof Cleaner" OOME exception 
handling to the shape of code predating change [5].
- add handling of OOME in the Cleaner.clean() with updated test that 
checks the behaviour.

Regards, Peter


[1] https://bugs.openjdk.java.net/browse/JDK-8066859
[2] 
http://cr.openjdk.java.net/~plevart/jdk9-dev/OOMEInReferenceHandler/Investigation/webrev.04/
[3] https://bugs.openjdk.java.net/browse/JDK-8015434
[4] 
http://mail.openjdk.java.net/pipermail/core-libs-dev/2013-April/016510.html
[5] http://hg.openjdk.java.net/jdk9/dev/jdk/rev/d04102f69d46
[6] 
https://www.mail-archive.com/core-libs-dev%40openjdk.java.net/msg23596.html
[7] https://bugs.openjdk.java.net/browse/JDK-8055232