Analysis on JDK-8022321 java/lang/ref/OOMEInReferenceHandler.java fails intermittently

[srikalyan]

Hi David

On 1/15/14, 9:04 PM, David Holmes wrote:
> On 16/01/2014 10:19 AM, srikalyan chandrashekar wrote:
>> Hi Peter/David, we could finally get a trace of exception with fastdebug
>> build and ReferenceHandler modified (with runImpl() added and called
>> from run()). The logs, disassembled code is available in JIRA
>> <https://bugs.openjdk.java.net/browse/JDK-8022321> as attachments.
>
> All I can see is the log for the OOMECatchingTest program not one for 
> the actual ReferenceHandler ??
>
Please search for ReferenceHandler in the log.
>> Observations from the log:
>>
>> Root Cause:
>> 1) UncaughtException is being dispatched from Reference.java:143
>> 141                   Reference<Object> r;
>> 142                   synchronized (lock) {
>> 143                        if (pending != null) {
>> 144                            r = pending;
>> 145                            pending = r.discovered;
>> 146                            r.discovered = null;
>>
>> pending field in Reference is touched and updated by the collector, so
>> at line 143 when the execution context is in Reference handler there
>> might have been an Exception pending due to allocation done by collector
>> which causes ReferenceHandler thread to die.
>
> Sorry but the GC does not trigger asynchronous exceptions so this 
> explanation does not make any sense to me. What part of the log led 
> you to this conclusion?
------------------ Log Excerpt begins ------------------
Exception <a 'java/lang/OutOfMemoryError'> (0x00000000ff7808e8)
thrown 
[/home/srikalyc/work/ora2013/infracleanup/jdk8/hotspot/src/share/vm/gc_interface/collectedHeap.inline.hpp, 
line 168]
for thread 0x00007feed80cf800
Exception <a 'java/lang/OutOfMemoryError'> (0x00000000ff7808e8)
  thrown in interpreter method <{method} {0x00007feeddd3c600} 'runImpl' 
'()V' in 'java/lang/ref/Reference$ReferenceHandler'>
  at bci 65 for thread 0x00007feed80cf800
Exception <a 'java/lang/OutOfMemoryError'> (0x00000000ff7808e8)
  thrown in interpreter method <{method} {0x00007feeddd3c478} 'run' 
'()V' in 'java/lang/ref/Reference$ReferenceHandler'>
  at bci 1 for thread 0x00007feed80cf800
Exception <a 'java/lang/OutOfMemoryError'> (0x00000000ff780868)
thrown 
[/home/srikalyc/work/ora2013/infracleanup/jdk8/hotspot/src/share/vm/gc_interface/collectedHeap.inline.hpp, 
line 157]
for thread 0x00007feed80cf800
Exception <a 'java/lang/OutOfMemoryError'> (0x00000000ff780868)
  thrown in interpreter method <{method} {0x00007feeddcaaf90} 
'uncaughtException' '(Ljava/lang/Thread;Ljava/lang/Throwable;)V' in '>
  at bci 48 for thread 0x00007feed80cf800
Exception <a 'java/lang/OutOfMemoryError'> (0x00000000ff780868)
  thrown in interpreter method <{method} {0x00007feeddca7298} 
'dispatchUncaughtException' '(Ljava/lang/Throwable;)V' in 'java/lang/>
  at bci 6 for thread 0x00007feed80cf800
------------------ Log Excerpt ends ------------------
Sorry if it is a wrong understanding.
>
>> Suggested fix:
>> - As proposed earlier putting an outer guard(try-catch on OOME) in the
>> ReferenceHandler will fix the issue, if ReferenceHandler is considered
>> as part of the GC sub system then it should be alive even in the midst
>> of an OOME so i feel that the additional guard should be allowed,
>> however i might still be ignorant of vital implications.
>> - Apart from the above changes, Peter's suggestion to create and call a
>> private runImpl() from run() in ReferenceHandler makes sense to me.
>
> Why would we need this?
>
> David
> -----
>
>>
>> ---
>> Thanks
>> kalyan
>>
>> On 01/13/2014 03:57 PM, srikalyan wrote:
>>>
>>> On 1/11/14, 6:15 AM, Peter Levart wrote:
>>>>
>>>> On 01/10/2014 10:51 PM, srikalyan chandrashekar wrote:
>>>>> Hi Peter the version you provided ran indefinitely(i put a 10 minute
>>>>> timeout) and the program got interrupted(no error),
>>>>
>>>> Did you run it with or without fastedbug & -XX:+TraceExceptions ? If
>>>> with, it might be that fastdebug and/or -XX:+TraceExceptions changes
>>>> the execution a bit so that we can no longer reproduce the wrong
>>>> behaviour.
>>> With fastdebug & -XX:TraceExceptions. I will try combination of
>>> possible options(i.e without -XX:TraceEception on debug build etc) 
>>> soon.
>>>>
>>>>> even if there were to be an error you cannot print the "string" of
>>>>> thread to console(these have been attempted earlier).
>>>>
>>>> ...it has been attempted to print toString in uncaught exception
>>>> handler. At that time, the heap is still full. I'm printing it after
>>>> the GC has cleared the heap. You can try that it works by commenting
>>>> out the "try {" and corresponding "} catch (OOME x) {}" exception
>>>> handler...
>>> Since there is a GC call prior to printing string i will give that a
>>> shot with non-debug build.
>>>>
>>>>> - The test's running on interpreter mode, what i am watching for is
>>>>> one error with trace. Without fastdebug build and
>>>>> -XX:+TraceExceptions i am able to reproduce failure atleast 5
>>>>> failures out of 1000 runs but with fastdebug+Trace no luck
>>>>> yet(already past few 1000 runs).
>>>>
>>>> It might be interesting to try with fastebug build but without the
>>>> -XX:+TraceExceptions option to see what has an effect on it. It might
>>>> also be interesting to try the modified ReferenceHandler (the one
>>>> with private runImpl() method called from run()) and with normal
>>>> non-fastdebug JDK. This info might be useful when one starts to
>>>> inspect the exception handling code in interpreter...
>>>>
>>>> Regards, Peter
>>>>
>>>
>>> -- 
>>> Thanks
>>> kalyan
>>> Ph: (408)-585-8040
>>>
>>>>>
>>>>> ---
>>>>> Thanks
>>>>> kalyan
>>>>>
>>>>> On 01/10/2014 02:57 AM, Peter Levart wrote:
>>>>>> On 01/10/2014 09:31 AM, Peter Levart wrote:
>>>>>>> Since we suspect there's something wrong with exception handling
>>>>>>> in interpreter, I devised a hypothetical reproducer that tries to
>>>>>>> simulate ReferenceHandler in many aspects, but doesn't require to
>>>>>>> be a ReferenceHandler:
>>>>>>>
>>>>>>> http://cr.openjdk.java.net/~plevart/misc/OOME/OOMECatchingTest.java
>>>>>>>
>>>>>>> This is designed to run indefinitely and only terminate if/when
>>>>>>> thread dies. Could you run this program in the environment that
>>>>>>> causes the OOMEInReferenceHandler test to fail and see if it
>>>>>>> terminates?
>>>>>>
>>>>>> I forgot to mention that in order for this long-running program to
>>>>>> exhibit interpreter behaviour, it should be run with -Xint option.
>>>>>> So I suggest:
>>>>>>
>>>>>> -Xmx24M -XX:-UseTLAB -Xint
>>>>>>
>>>>>> Regards, Peter
>>>>>>
>>>>>
>>>>
>>