how to bring jvm to safepoint
Tony Guan
guanxiaohua at gmail.com
Tue Aug 3 16:20:26 PDT 2010
Hi David,
I changed dtrace_method_exit to JRT_ENTRY from JRT_LEAF, and still
stopped by the same failure. Does it have anything to do with the
method sun.misc.Unsafe.park()?
Thanks!
Tony (Xiaohua Guan)
On Sun, Aug 1, 2010 at 3:50 PM, Tony Guan <guanxiaohua at gmail.com> wrote:
> Hi David,
>
> Thanks for the explanation, now I get to understand it.
>
> Have to say, it's a nice failure for me. :)
>
> Tony (Xiaohua Guan)
>
>
>
> On Sat, Jul 31, 2010 at 10:43 PM, David Holmes <David.Holmes at oracle.com> wrote:
>> Hi Tony,
>>
>> Now it is all clear :)
>>
>>> It's exactly how the error occurred, I am monitoring the
>>> sun.misc.Unsafe.park() method call with dtrace(no dtrace agent, just
>>> intercepting inside jvm implementation), once this method gets called,
>>> I will call GenCollectedHeap::invoke_transEden_collection() to start a
>>> GC using VMThread::Execute(&op).
>>
>> Right, you are using:
>>
>> _ZN13SharedRuntime18dtrace_method_exitEP10JavaThreadP13methodOopDesc+0x255
>>
>> but dtrace_method_exit is defined as a runtime "leaf" method (JRT_LEAF).
>> "leaf" methods are not allowed to perform certain actions and in debug mode
>> we have JRT_Leaf_Verifier that is used to check those actions. A
>> JRT_Leaf_Verifier is a No_Safepoint_Verifier - hence as I said before you
>> get the assertion failure due to this.
>>
>> When you do the VMThread::execute to initiate the safepoint operation for
>> your GC action, the current thread will block waiting for it to complete.
>> Part of that blocking involves a safepoint check and hence the system finds
>> that the thread is attempting to enter a safepoint from a "leaf" method -
>> and that is not allowed.
>>
>> You might just try changing the definition of dtrace_method_exit to be
>> JRT_ENTRY rather than JRT_LEAF, but I don't know if that will have
>> unexpected side-effects.
>>
>> There are lots of rules (not necessarily clearly documented) that control
>> how you can call into the VM runtime and how different parts of the VM
>> runtime can call other parts of the runtime.
>>
>> Cheers,
>> David Holmes
>>
>> Tony Guan said the following on 08/01/10 08:10:
>>>
>>> Hi David & Tom,
>>>
>>> Please see the decoded stack below(also full decoded log is attached):
>>>
>>> "
>>> Current thread (0x00000000012ed000): JavaThread "RMI TCP
>>> Connection(idle)" daemon [_thread_in_Java, id=30929,
>>> stack(0x000000004073b000,0x000000004083c000)]
>>>
>>> Stack: [0x000000004073b000,0x000000004083c000],
>>> sp=0x0000000040839d60, free space=3fb0000000000000000k
>>> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
>>> code)
>>> V [libjvm.so+0x8cad18];; _ZN7VMError6reportEP12outputStream+0xb72
>>> V [libjvm.so+0x8cbe38];; _ZN7VMError14report_and_dieEv+0x5f6
>>> V [libjvm.so+0x4039f1];; _Z12report_fatalPKciS0_+0x6b
>>> V [libjvm.so+0x87a183];;
>>> _ZN6Thread31check_for_valid_safepoint_stateEb+0x33
>>> V [libjvm.so+0x8e1127];; _ZN8VMThread7executeEP12VM_Operation+0x49
>>> V [libjvm.so+0x4d52ab];;
>>> _ZN16GenCollectedHeap27invoke_transEden_collectionEi+0x33
>>> V [libjvm.so+0x7ee63d];;
>>> _ZN13SharedRuntime18dtrace_method_exitEP10JavaThreadP13methodOopDesc+0x255
>>> j sun.misc.Unsafe.park(ZJ)V+0
>>> j
>>> java.util.concurrent.locks.LockSupport.parkNanos(Ljava/lang/Object;J)V+20
>>> j
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Ljava/util/concurrent/SynchronousQueue$TransferStack$SNode;ZJ)Ljava/util/concurrent/SynchronousQueue$TransferStack$SNode;+174
>>> j
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(Ljava/lang/Object;ZJ)Ljava/lang/Object;+102
>>> j
>>> java.util.concurrent.SynchronousQueue.poll(JLjava/util/concurrent/TimeUnit;)Ljava/lang/Object;+11
>>> j
>>> java.util.concurrent.ThreadPoolExecutor.getTask()Ljava/lang/Runnable;+141
>>> j
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+17
>>> j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
>>> j java.lang.Thread.run()V+11
>>> v ~StubRoutines::call_stub
>>> V [libjvm.so+0x555ecc];;
>>>
>>> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread+0x474
>>> V [libjvm.so+0x747a34];;
>>>
>>> _ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_+0x32
>>> V [libjvm.so+0x554b55];;
>>>
>>> _ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread+0x6f
>>> V [libjvm.so+0x5551f0];;
>>>
>>> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread+0x158
>>> V [libjvm.so+0x5553cc];;
>>>
>>> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread+0x86
>>> V [libjvm.so+0x5be6cd];; _Z12thread_entryP10JavaThreadP6Thread+0x9f
>>> V [libjvm.so+0x879869];; _ZN10JavaThread17thread_main_innerEv+0xc9
>>> V [libjvm.so+0x87c1b5];; _ZN10JavaThread3runEv+0x2c1
>>> V [libjvm.so+0x750455];; _Z10java_startP6Thread+0x16f
>>> "
>>>
>>> It's exactly how the error occurred, I am monitoring the
>>> sun.misc.Unsafe.park() method call with dtrace(no dtrace agent, just
>>> intercepting inside jvm implementation), once this method gets called,
>>> I will call GenCollectedHeap::invoke_transEden_collection() to start a
>>> GC using VMThread::Execute(&op).
>>>
>>> At the beginning of VMTHread::Execute(),
>>> check_for_valid_safepoint_state is called like this:
>>>
>>> void VMThread::execute(VM_Operation* op) {
>>> Thread* t = Thread::current();
>>> if (!t->is_VM_thread()) {
>>> // JavaThread or WatcherThread
>>> t->check_for_valid_safepoint_state(true); //And we failed here!!
>>> // New request from Java thread, evaluate prologue
>>> if (!op->doit_prologue()) {
>>> return; // op was cancelled
>>> }
>>> ...
>>> }
>>>
>>> That's why I said the current thread is not a VM_thread. I think I can
>>> make a workaround by invoking the GC at the next allocation request,
>>> which should be in the safepoint. But would be great if I can learn
>>> more about hotspot solving this problem.
>>>
>>> Thanks a lot!
>>>
>>> Tony (Xiaohua Guan)
>>>
>>>
>>>
>>> On Sat, Jul 31, 2010 at 12:56 AM, David Holmes <David.Holmes at oracle.com>
>>> wrote:
>>>>
>>>> Tom Rodriguez said the following on 07/31/10 05:01:
>>>>>
>>>>> Actually I think the problem is that the thread is in thread_in_Java
>>>>> state. I don't believe you can safepoint from that state, only
>>>>> thread_in_vm.
>>>>
>>>> The reported failure:
>>>>
>>>> # Error: Possible safepoint reached by thread that does not allow it
>>>>
>>>> is here:
>>>>
>>>> void Thread::check_for_valid_safepoint_state(bool potential_vm_operation)
>>>> {
>>>> // Check if current thread is allowed to block at a safepoint
>>>> if (!(_allow_safepoint_count == 0))
>>>> fatal("Possible safepoint reached by thread that does not allow it");
>>>>
>>>> as far as I can see the only code that touches _allow_safepoint_count are
>>>> the No_Safepoint_Verifier classes.
>>>>
>>>> Also the stack shows:
>>>>
>>>> V [libjvm.so+0x7ee63d]
>>>> j sun.misc.Unsafe.park(ZJ)V+0
>>>>
>>>> and so we should have transitioned to _thread_in_vm here.
>>>>
>>>> That said, Tony also stated:
>>>>
>>>>> The reason for the fail is that
>>>>> Thread::check_for_valid_safepoint_state() didn't pass because
>>>>> the currentThread is not a VMThread.
>>>>
>>>> But that is not supported by the error produced.
>>>>
>>>> Tony: why did you make the above comment?
>>>>
>>>> Either way we need to see a decoded stack to get an idea of what is going
>>>> wrong.
>>>>
>>>> Cheers,
>>>> David
>>>>
>>>>> tom
>>>>>
>>>>> On Jul 30, 2010, at 12:04 AM, David Holmes <David.Holmes at oracle.com>
>>>>> wrote:
>>>>>
>>>>>> Sorry Tony I keep forgetting that I don't have a way to decode a
>>>>>> crash log from a VM that you built yourself. :( However you may be
>>>>>> able to decode it yourself please see:
>>>>>>
>>>>>> http://blogs.sun.com/dave/entry/a_tool_to_decode_hs
>>>>>>
>>>>>> for a perl script.
>>>>>>
>>>>>> That said looking at the actual assertion failure I am guessing
>>>>>> that you initiated the VM operation from code where a
>>>>>> No_Safepoint_verifier is active.
>>>>>>
>>>>>> David
>>>>>>
>>>>>> Tony Guan said the following on 07/30/10 15:58:
>>>>>>>
>>>>>>> Hi David, Thanks a lot! I am copying the content to below, and
>>>>>>> for convenience, the full log file is attached. cat
>>>>>>> hs_err_pid30894.log # # A fatal error has been detected by the
>>>>>>> Java Runtime Environment: # # Internal Error
>>>>>>>
>>>>>>> (/home/tony/software/OpenJDK/jdk7/hotspot/src/share/vm/runtime/thread.cpp:777),
>>>>>>> pid=30894, tid=1082374480 # Error: Possible safepoint reached
>>>>>>> by thread that does not allow it # # JRE version: 7.0 # Java VM:
>>>>>>> OpenJDK 64-Bit Server VM (17.0-b07291505-internal-debug mixed
>>>>>>> mode linux-amd64 ) # If you would like to submit a bug report,
>>>>>>> please visit: # http://java.sun.com/webapps/bugreport/crash.jsp
>>>>>>> # --------------- T H R E A D --------------- Current thread
>>>>>>> (0x00000000012ed000): JavaThread "RMI TCP Connection(idle)"
>>>>>>> daemon [_thread_in_Java, id=30929,
>>>>>>> stack(0x000000004073b000,0x000000004083c000)] Stack:
>>>>>>> [0x000000004073b000,0x000000004083c000], sp=0x0000000040839d60,
>>>>>>> free space=3fb0000000000000000k Native frames: (J=compiled Java
>>>>>>> code, j=interpreted, Vv=VM code, C=native code) V
>>>>>>> [libjvm.so+0x8cad18] V [libjvm.so+0x8cbe38] V
>>>>>>> [libjvm.so+0x4039f1] V [libjvm.so+0x87a183] V
>>>>>>> [libjvm.so+0x8e1127] V [libjvm.so+0x4d52ab] V
>>>>>>> [libjvm.so+0x7ee63d] j sun.misc.Unsafe.park(ZJ)V+0 j
>>>>>>>
>>>>>>>
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(Ljava/lang/Object;J)V+20
>>>>>>> j
>>>>>>>
>>>>>>>
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Ljava/util/concurrent/SynchronousQueue$TransferStack$SNode;ZJ)Ljava/util/concurrent/SynchronousQueue$TransferStack$SNode;+174
>>>>>>> j
>>>>>>>
>>>>>>>
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(Ljava/lang/Object;ZJ)Ljava/lang/Object;+102
>>>>>>> j
>>>>>>>
>>>>>>>
>>>>>>> java.util.concurrent.SynchronousQueue.poll(JLjava/util/concurrent/TimeUnit;)Ljava/lang/Object;+11
>>>>>>> j
>>>>>>>
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask()Ljava/lang/Runnable;+141
>>>>>>> j
>>>>>>>
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+17
>>>>>>> j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 j
>>>>>>> java.lang.Thread.run()V+11 v ~StubRoutines::call_stub V
>>>>>>> [libjvm.so+0x555ecc] V [libjvm.so+0x747a34] V
>>>>>>> [libjvm.so+0x554b55] V [libjvm.so+0x5551f0] V
>>>>>>> [libjvm.so+0x5553cc] V [libjvm.so+0x5be6cd] V
>>>>>>> [libjvm.so+0x879869] V [libjvm.so+0x87c1b5] V
>>>>>>> [libjvm.so+0x750455] .... (above is the stack, please see the
>>>>>>> attachment for more information) Tony (Xiaohua Guan) On Thu, Jul
>>>>>>> 29, 2010 at 8:07 PM, David Holmes <David.Holmes at oracle.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Tony,
>>>>>>>>
>>>>>>>> I don't see anything obviously wrong with what you attempted.
>>>>>>>> Can you show the full stack dump from the error.
>>>>>>>>
>>>>>>>> David Holmes
>>>>>>>>
>>>>>>>> Tony Guan said the following on 07/30/10 07:29:
>>>>>>>>>
>>>>>>>>> Dear all,
>>>>>>>>>
>>>>>>>>> I want to invoke the GC at a certain time, for example, when
>>>>>>>>> a certain method is called at runtime, so I created a new
>>>>>>>>> subtype of VM_GC_Operation class, and initialized it with an
>>>>>>>>> object op, finally, executed using:VMThread::execute(&op)
>>>>>>>>>
>>>>>>>>> then I got the following error message:
>>>>>>>>>
>>>>>>>>> # Internal Error
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> (/home/tony/software/OpenJDK/jdk7/hotspot/src/share/vm/runtime/thread.cpp:777),
>>>>>>>>> pid=29652, tid=1096460624 # Error: Possible safepoint
>>>>>>>>> reached by thread that does not allow it
>>>>>>>>>
>>>>>>>>> The reason for the fail is that
>>>>>>>>> Thread::check_for_valid_safepoint_state() didn't pass because
>>>>>>>>> the currentThread is not a VMThread.
>>>>>>>>>
>>>>>>>>> My question is: how do I bring the current thread to a
>>>>>>>>> safepoint? or how do I call a GC operation properly? is there
>>>>>>>>> any prerequisite for doing so?
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>> Tony (Xiaohua Guan)
>>
>
More information about the hotspot-runtime-dev
mailing list