Analysis on JDK-8022321 java/lang/ref/OOMEInReferenceHandler.java fails intermittently

[David Holmes]

Hi Peter,

On 17/01/2014 11:24 PM, Peter Levart wrote:
> On 01/17/2014 02:13 PM, Peter Levart wrote:
>>>> // Fast path for cleaners
>>>> boolean isCleaner = false;
>>>> try {
>>>>   isCleaner = r instanceof Cleaner;
>>>> } catch (OutofMemoryError oome) {
>>>>   continue;
>>>> }
>>>>
>>>> if (isCleaner) {
>>>>   ((Cleaner)r).clean();
>>>>   continue;
>>>> }
>>>>
>>>
>>> Hi David, Kalyan,
>>>
>>> I've caught-up now. Just thinking: is "instanceof Cleaner" throwing
>>> OOME as a result of loading the Cleaner class? Wouldn't the above
>>> code then throw some error also in ((Cleaner)r) - the checkcast,
>>> since Cleaner class would not be successfully initialized?
>>
>> Well, no. The above code would just skip Cleaner processing in this
>> situation. And will never be doing it again after the heap is freed...
>> So it might be good to load and initialize Cleaner class as part of
>> ReferenceHandler initialization to ensure correct operation...
>
> Well, yes and no. Let me try once more:
>
> Above code will skip Cleaner processing if the 1st time "instanceof
> Cleaner" is executed, OOME is thrown as a consequence of full heap while
> loading and initializing the Cleaner class.

Yes - I was assuming that this would not fail the very first time and so 
the Cleaner class would already be loaded. Failing to be able to load 
the Cleaner class was one of the potential issues flagged earlier with 
this problem. I was actually assuming that Cleaner would be loaded 
already due to some actual Cleaner subclasses being used, but this does 
not happen as part of the default initialization. :( The irony being 
that if the Cleaner class is not loaded then r can not be an instance of 
Cleaner and so we would fail to load the class in a case where we didn't 
need it anyway.

What I wanted to focus on here was an OOME from the instanceof itself, 
but as you say that might trigger classloading of Cleaner (which is not 
what I was interested in).

> The 2nd time the "instanceof
> Cleaner" is executed after such OOME, the same line would throw
> NoClassDefFoundError as a consequence of referencing a class that failed
> initialization. Am I right?

instanceof is not one of the class initialization triggers, so we should 
not see an OOME generated due to a class initialization exception and so 
the class will not be put into the Erroneous state and so subsequent 
attempts to use the class will not automatically trigger 
NoClassdefFoundError.

If OOME occurs during actual loading/linking of the class Cleaner it is 
unclear what would happen on subsequent attempts. OOME is not a 
LinkageError that must be rethrown on subsequent attempts, and it is 
potentially a transient condition, so I would expect a re-load attempt 
to be allowed. However we are now deep into the details of the VM and it 
may well depend on the exact place from which the OOME originates.

The bottom line with the current problem is that there are multiple 
non-obvious paths by which the ReferenceHandler can encounter an OOME. 
In such cases we do not want the ReferenceHandler to terminate - which 
implies catching the OOME and continuing. However we also do not want to 
silently skip Cleaner processing or reference queue processing - as that 
would lead to hard to diagnoze bugs. But trying to report the problem 
may not be possible due to being out-of-memory. It may be that we need 
to break things up into multiple try/catch blocks, where each catch does 
a System.gc() and then reports that the OOME occurred. Of course the 
reporting must still be in a try/catch for the OOME. Though at some 
point letting the ReferenceHandler die may be the only way to "report" a 
major memory problem.

David

David

> Regards, Peter
>