how to bring jvm to safepoint

Tony Guan guanxiaohua at gmail.com
Tue Aug 3 16:38:32 PDT 2010 

Hi Everyone,

Please ignore my previous message, I just forgot to update the jvm for testing.

Actually, changing dtrace_method_exit to be JRT_ENTRY solved the
problem! It works out.

Thank you, David!


Cheers!

Tony (Xiaohua Guan)



On Tue, Aug 3, 2010 at 6:20 PM, Tony Guan <guanxiaohua at gmail.com> wrote:
> Hi David,
>
> I changed dtrace_method_exit to JRT_ENTRY from JRT_LEAF, and still
> stopped by the same failure. Does it have anything to do with the
> method sun.misc.Unsafe.park()?
>
> Thanks!
>
> Tony (Xiaohua Guan)
>
>
>
> On Sun, Aug 1, 2010 at 3:50 PM, Tony Guan <guanxiaohua at gmail.com> wrote:
>> Hi David,
>>
>> Thanks for the explanation, now I get to understand it.
>>
>> Have to say, it's a nice failure for me. :)
>>
>> Tony (Xiaohua Guan)
>>
>>
>>
>> On Sat, Jul 31, 2010 at 10:43 PM, David Holmes <David.Holmes at oracle.com> wrote:
>>> Hi Tony,
>>>
>>> Now it is all clear :)
>>>
>>>> It's exactly how the error occurred, I am monitoring the
>>>> sun.misc.Unsafe.park() method call with dtrace(no dtrace agent, just
>>>> intercepting inside jvm implementation), once this method gets called,
>>>> I will call GenCollectedHeap::invoke_transEden_collection() to start a
>>>> GC using VMThread::Execute(&op).
>>>
>>> Right, you are using:
>>>
>>> _ZN13SharedRuntime18dtrace_method_exitEP10JavaThreadP13methodOopDesc+0x255
>>>
>>> but dtrace_method_exit is defined as a runtime "leaf" method (JRT_LEAF).
>>> "leaf" methods are not allowed to perform certain actions and in debug mode
>>> we have JRT_Leaf_Verifier that is used to check those actions. A
>>> JRT_Leaf_Verifier is a No_Safepoint_Verifier - hence as I said before you
>>> get the assertion failure due to this.
>>>
>>> When you do the VMThread::execute to initiate the safepoint operation for
>>> your GC action, the current thread will block waiting for it to complete.
>>> Part of that blocking involves a safepoint check and hence the system finds
>>> that the thread is attempting to enter a safepoint from a "leaf" method -
>>> and that is not allowed.
>>>
>>> You might just try changing the definition of dtrace_method_exit to be
>>> JRT_ENTRY rather than JRT_LEAF, but I don't know if that will have
>>> unexpected side-effects.
>>>
>>> There are lots of rules (not necessarily clearly documented) that control
>>> how you can call into the VM runtime and how different parts of the VM
>>> runtime can call other parts of the runtime.
>>>
>>> Cheers,
>>> David Holmes
>>>
>>> Tony Guan said the following on 08/01/10 08:10:
>>>>
>>>> Hi David & Tom,
>>>>
>>>> Please see the decoded stack below(also full decoded log is attached):
>>>>
>>>> "
>>>> Current thread (0x00000000012ed000):  JavaThread "RMI TCP
>>>> Connection(idle)" daemon [_thread_in_Java, id=30929,
>>>> stack(0x000000004073b000,0x000000004083c000)]
>>>>
>>>> Stack: [0x000000004073b000,0x000000004083c000],
>>>> sp=0x0000000040839d60,  free space=3fb0000000000000000k
>>>> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
>>>> code)
>>>> V  [libjvm.so+0x8cad18];;  _ZN7VMError6reportEP12outputStream+0xb72
>>>> V  [libjvm.so+0x8cbe38];;  _ZN7VMError14report_and_dieEv+0x5f6
>>>> V  [libjvm.so+0x4039f1];;  _Z12report_fatalPKciS0_+0x6b
>>>> V  [libjvm.so+0x87a183];;
>>>>  _ZN6Thread31check_for_valid_safepoint_stateEb+0x33
>>>> V  [libjvm.so+0x8e1127];;  _ZN8VMThread7executeEP12VM_Operation+0x49
>>>> V  [libjvm.so+0x4d52ab];;
>>>> _ZN16GenCollectedHeap27invoke_transEden_collectionEi+0x33
>>>> V  [libjvm.so+0x7ee63d];;
>>>> _ZN13SharedRuntime18dtrace_method_exitEP10JavaThreadP13methodOopDesc+0x255
>>>> j  sun.misc.Unsafe.park(ZJ)V+0
>>>> j
>>>>  java.util.concurrent.locks.LockSupport.parkNanos(Ljava/lang/Object;J)V+20
>>>> j
>>>>  java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Ljava/util/concurrent/SynchronousQueue$TransferStack$SNode;ZJ)Ljava/util/concurrent/SynchronousQueue$TransferStack$SNode;+174
>>>> j
>>>>  java.util.concurrent.SynchronousQueue$TransferStack.transfer(Ljava/lang/Object;ZJ)Ljava/lang/Object;+102
>>>> j
>>>>  java.util.concurrent.SynchronousQueue.poll(JLjava/util/concurrent/TimeUnit;)Ljava/lang/Object;+11
>>>> j
>>>>  java.util.concurrent.ThreadPoolExecutor.getTask()Ljava/lang/Runnable;+141
>>>> j
>>>>  java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+17
>>>> j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
>>>> j  java.lang.Thread.run()V+11
>>>> v  ~StubRoutines::call_stub
>>>> V  [libjvm.so+0x555ecc];;
>>>>
>>>> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread+0x474
>>>> V  [libjvm.so+0x747a34];;
>>>>
>>>> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_+0x32
>>>> V  [libjvm.so+0x554b55];;
>>>>
>>>> _ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread+0x6f
>>>> V  [libjvm.so+0x5551f0];;
>>>>
>>>> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread+0x158
>>>> V  [libjvm.so+0x5553cc];;
>>>>
>>>> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread+0x86
>>>> V  [libjvm.so+0x5be6cd];;  _Z12thread_entryP10JavaThreadP6Thread+0x9f
>>>> V  [libjvm.so+0x879869];;  _ZN10JavaThread17thread_main_innerEv+0xc9
>>>> V  [libjvm.so+0x87c1b5];;  _ZN10JavaThread3runEv+0x2c1
>>>> V  [libjvm.so+0x750455];;  _Z10java_startP6Thread+0x16f
>>>> "
>>>>
>>>> It's exactly how the error occurred, I am monitoring the
>>>> sun.misc.Unsafe.park() method call with dtrace(no dtrace agent, just
>>>> intercepting inside jvm implementation), once this method gets called,
>>>> I will call GenCollectedHeap::invoke_transEden_collection() to start a
>>>> GC using VMThread::Execute(&op).
>>>>
>>>> At the beginning of VMTHread::Execute(),
>>>> check_for_valid_safepoint_state is called like this:
>>>>
>>>> void VMThread::execute(VM_Operation* op) {
>>>>  Thread* t = Thread::current();
>>>>  if (!t->is_VM_thread()) {
>>>>    // JavaThread or WatcherThread
>>>>    t->check_for_valid_safepoint_state(true);  //And we failed here!!
>>>>    // New request from Java thread, evaluate prologue
>>>>    if (!op->doit_prologue()) {
>>>>      return;   // op was cancelled
>>>>    }
>>>> ...
>>>> }
>>>>
>>>> That's why I said the current thread is not a VM_thread. I think I can
>>>> make a workaround by invoking the GC at the next allocation request,
>>>> which should be in the safepoint. But would be great if I can learn
>>>> more about hotspot solving this problem.
>>>>
>>>> Thanks a lot!
>>>>
>>>> Tony (Xiaohua Guan)
>>>>
>>>>
>>>>
>>>> On Sat, Jul 31, 2010 at 12:56 AM, David Holmes <David.Holmes at oracle.com>
>>>> wrote:
>>>>>
>>>>> Tom Rodriguez said the following on 07/31/10 05:01:
>>>>>>
>>>>>> Actually I think the problem is that the thread is in thread_in_Java
>>>>>> state. I don't believe you can safepoint from that state, only
>>>>>> thread_in_vm.
>>>>>
>>>>> The reported failure:
>>>>>
>>>>> #  Error: Possible safepoint reached by thread that does not allow it
>>>>>
>>>>> is here:
>>>>>
>>>>> void Thread::check_for_valid_safepoint_state(bool potential_vm_operation)
>>>>> {
>>>>>   // Check if current thread is allowed to block at a safepoint
>>>>>   if (!(_allow_safepoint_count == 0))
>>>>>     fatal("Possible safepoint reached by thread that does not allow it");
>>>>>
>>>>> as far as I can see the only code that touches _allow_safepoint_count are
>>>>> the No_Safepoint_Verifier classes.
>>>>>
>>>>> Also the stack shows:
>>>>>
>>>>> V  [libjvm.so+0x7ee63d]
>>>>> j  sun.misc.Unsafe.park(ZJ)V+0
>>>>>
>>>>> and so we should have transitioned to _thread_in_vm here.
>>>>>
>>>>> That said, Tony also stated:
>>>>>
>>>>>> The reason for the fail is that
>>>>>> Thread::check_for_valid_safepoint_state() didn't pass because
>>>>>> the currentThread is not a VMThread.
>>>>>
>>>>> But that is not supported by the error produced.
>>>>>
>>>>> Tony: why did you make the above comment?
>>>>>
>>>>> Either way we need to see a decoded stack to get an idea of what is going
>>>>> wrong.
>>>>>
>>>>> Cheers,
>>>>> David
>>>>>
>>>>>> tom
>>>>>>
>>>>>> On Jul 30, 2010, at 12:04 AM, David Holmes <David.Holmes at oracle.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Sorry Tony I keep forgetting that I don't have a way to decode a
>>>>>>> crash log from a VM that you built yourself. :( However you may be
>>>>>>> able to decode it yourself please see:
>>>>>>>
>>>>>>> http://blogs.sun.com/dave/entry/a_tool_to_decode_hs
>>>>>>>
>>>>>>> for a perl script.
>>>>>>>
>>>>>>> That said looking at the actual assertion failure I am guessing
>>>>>>> that you initiated the VM operation from code where a
>>>>>>> No_Safepoint_verifier is active.
>>>>>>>
>>>>>>> David
>>>>>>>
>>>>>>> Tony Guan said the following on 07/30/10 15:58:
>>>>>>>>
>>>>>>>> Hi David, Thanks a lot! I am copying the content to below, and
>>>>>>>> for convenience, the full log file is attached. cat
>>>>>>>> hs_err_pid30894.log # # A fatal error has been detected by the
>>>>>>>> Java Runtime Environment: # #  Internal Error
>>>>>>>>
>>>>>>>> (/home/tony/software/OpenJDK/jdk7/hotspot/src/share/vm/runtime/thread.cpp:777),
>>>>>>>>  pid=30894, tid=1082374480 #  Error: Possible safepoint reached
>>>>>>>> by thread that does not allow it # # JRE version: 7.0 # Java VM:
>>>>>>>> OpenJDK 64-Bit Server VM (17.0-b07291505-internal-debug mixed
>>>>>>>> mode linux-amd64 ) # If you would like to submit a bug report,
>>>>>>>> please visit: #   http://java.sun.com/webapps/bugreport/crash.jsp
>>>>>>>>  # ---------------  T H R E A D  --------------- Current thread
>>>>>>>> (0x00000000012ed000):  JavaThread "RMI TCP Connection(idle)"
>>>>>>>> daemon [_thread_in_Java, id=30929,
>>>>>>>> stack(0x000000004073b000,0x000000004083c000)] Stack:
>>>>>>>> [0x000000004073b000,0x000000004083c000], sp=0x0000000040839d60,
>>>>>>>> free space=3fb0000000000000000k Native frames: (J=compiled Java
>>>>>>>> code, j=interpreted, Vv=VM code, C=native code) V
>>>>>>>> [libjvm.so+0x8cad18] V  [libjvm.so+0x8cbe38] V
>>>>>>>> [libjvm.so+0x4039f1] V  [libjvm.so+0x87a183] V
>>>>>>>> [libjvm.so+0x8e1127] V  [libjvm.so+0x4d52ab] V
>>>>>>>> [libjvm.so+0x7ee63d] j  sun.misc.Unsafe.park(ZJ)V+0 j
>>>>>>>>
>>>>>>>>
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(Ljava/lang/Object;J)V+20
>>>>>>>>  j
>>>>>>>>
>>>>>>>>
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Ljava/util/concurrent/SynchronousQueue$TransferStack$SNode;ZJ)Ljava/util/concurrent/SynchronousQueue$TransferStack$SNode;+174
>>>>>>>>  j
>>>>>>>>
>>>>>>>>
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(Ljava/lang/Object;ZJ)Ljava/lang/Object;+102
>>>>>>>>  j
>>>>>>>>
>>>>>>>>
>>>>>>>> java.util.concurrent.SynchronousQueue.poll(JLjava/util/concurrent/TimeUnit;)Ljava/lang/Object;+11
>>>>>>>>  j
>>>>>>>>
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask()Ljava/lang/Runnable;+141
>>>>>>>>  j
>>>>>>>>
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+17
>>>>>>>>  j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 j
>>>>>>>> java.lang.Thread.run()V+11 v  ~StubRoutines::call_stub V
>>>>>>>> [libjvm.so+0x555ecc] V  [libjvm.so+0x747a34] V
>>>>>>>> [libjvm.so+0x554b55] V  [libjvm.so+0x5551f0] V
>>>>>>>> [libjvm.so+0x5553cc] V  [libjvm.so+0x5be6cd] V
>>>>>>>> [libjvm.so+0x879869] V  [libjvm.so+0x87c1b5] V
>>>>>>>> [libjvm.so+0x750455] .... (above is the stack, please see the
>>>>>>>> attachment for more information) Tony (Xiaohua Guan) On Thu, Jul
>>>>>>>> 29, 2010 at 8:07 PM, David Holmes <David.Holmes at oracle.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Tony,
>>>>>>>>>
>>>>>>>>> I don't see anything obviously wrong with what you attempted.
>>>>>>>>> Can you show the full stack dump from the error.
>>>>>>>>>
>>>>>>>>> David Holmes
>>>>>>>>>
>>>>>>>>> Tony Guan said the following on 07/30/10 07:29:
>>>>>>>>>>
>>>>>>>>>> Dear all,
>>>>>>>>>>
>>>>>>>>>> I want to invoke the GC at a certain time, for example, when
>>>>>>>>>> a certain method is called at runtime, so I created a new
>>>>>>>>>> subtype of VM_GC_Operation class, and initialized it with an
>>>>>>>>>> object op, finally, executed using:VMThread::execute(&op)
>>>>>>>>>>
>>>>>>>>>> then I got the following error message:
>>>>>>>>>>
>>>>>>>>>> #  Internal Error
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> (/home/tony/software/OpenJDK/jdk7/hotspot/src/share/vm/runtime/thread.cpp:777),
>>>>>>>>>>  pid=29652, tid=1096460624 #  Error: Possible safepoint
>>>>>>>>>> reached by thread that does not allow it
>>>>>>>>>>
>>>>>>>>>> The reason for the fail is that
>>>>>>>>>> Thread::check_for_valid_safepoint_state() didn't pass because
>>>>>>>>>> the currentThread is not a VMThread.
>>>>>>>>>>
>>>>>>>>>> My question is: how do I bring the current thread to a
>>>>>>>>>> safepoint? or how do I call a GC operation properly? is there
>>>>>>>>>> any prerequisite for doing so?
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>>
>>>>>>>>>> Tony (Xiaohua Guan)
>>>
>>
>

More information about the hotspot-runtime-dev mailing list