Safepoint Deadlock Diagnosis

[Peng Du]

David,

I found where the livelock was resulted. When methodOopDesc::_code was 
cleared, my
code would cause _from_compiled_entry uncleared, thus 
handle_wrong_method tries to
jumps to _from_compiled_entry, thus causing another resolution. This 
goes on forever.
Thus, seems like a livelock. It's totally my fault.

Thanks, guys

-Peng


On 04/13/2010 05:53 PM, David Holmes wrote:
> I'd suggest turning on all class-loading/resolution related tracing. 
> Seems to me that deopt is forcing a call-site to be re-resolved while 
> a safepoint is pending. But the re-resolution seems to be livelocking 
> somewhere.
>
> David
>
> Peng Du said the following on 04/14/10 08:42:
>> Thanks, Ramki and David
>>
>> Sorry for the confusion, I was running a custom HotSpot that I have 
>> modified.
>> So it's definitely my fault :)
>>
>> After the previous post, I did more investigations by trying the 
>> various -XX
>> options, especially those on safepoints, e.g. TraceSafepoint, 
>> ShowSafepointMsgs,
>> and etc. I saw lots of consecutive:
>>
>> ======
>> Safepoint synchronization initiated. (7)
>> Entering safepoint region: ForceSafepoint
>> do_cleanup_tasks takes      0ms
>> Leaving safepoint region
>> ======
>>
>> I am not sure if that was the normal behavior. Then, I tried: 
>> TraceRuntimeCalls,
>> which confirmed that HotSpot kept spinning on 
>> SharedRuntime::handle_wrong_method
>> a large number of times. So, I guess David is right. It seems like a 
>> live lock.
>>
>> Can someone explain why would HotSpot keep calling 
>> handle_wrong_method()?
>>
>> Thanks
>>
>>
>> Peng
>>
>>
>>
>> On 04/13/2010 05:15 PM, Y. Srinivas Ramakrishna wrote:
>>> Try the debug jvm if not already doing so and it might tell you
>>> why you are going to deadlock. If not, then you might have some
>>> other kind of class loading deadlock which you can learn more about
>>> from the class loading cognoscenti on this list. (May be use
>>> TraceLoaderConstraints and TraceClass* flags for more visibility
>>> into the sequence of events leading to this.) Since deadlocks
>>> involve more than a single thread, you might have wanted to
>>> look at other threads to see where they were. It wasn't clear
>>> where this thread was stuck (was it acquiring a lock or waiting
>>> for notification on a cond_var)? You might be able to trace the
>>> waits-for chain on the JVM locks at issue here to understand
>>> the nature of the deadlock if any.
>>>
>>> -- ramki
>>>
>>> On 04/13/10 13:40, Peng Du wrote:
>>>> Hello, list
>>>>
>>>> I have been witnessing deadlocks of the HotSpot I am working on. I 
>>>> tried everything I know
>>>> to diagnose this. Here is what I found: the "main" JavaThread 
>>>> always ended up somewhere
>>>> in the stub SharedRuntime::handle_wrong_method(). HotSpot would 
>>>> then deadlock at this point.
>>>> In to the thread dump, I saw all threads are: _thread_blocked. The 
>>>> method that was patched
>>>> with this stub is "java_lang_math_log" (frame 12 @ 
>>>> 0x00007f832173b40e).
>>>>
>>>> I have attached the backtrace and thread dump in the email. Can 
>>>> someone hint on what might
>>>> be the reason for this deadlock?
>>>>
>>>> Thanks!
>>>>
>>>>
>>>> -Peng
>>>>
>>>
>>