For-Test: 8212933: Thread-SMR: requesting a VM operation whilst holding a ThreadsListHandle can cause deadlocks

Robbin Ehn robbin.ehn at oracle.com
Thu Oct 25 13:15:42 UTC 2018


Hi, here is a fix, which allows ThreadsList to be used over a VM operation.

http://cr.openjdk.java.net/~rehn/8212933/v1_handshak_vm_cancel/webrev/

Please test it out.

/Robbin

On 10/24/18 11:16 PM, serguei.spitsyn at oracle.com wrote:
> Leonid confirmed this deadlock is not reproducible if the Kitchensink 
> agent_sampler is disabled.
> Also, applying the patch from Robbin (with agent_sampler enabled) hit new assert
> that has caught another case in JvmtiEnv::GetStackTrace with the same pattern:
> 
> With proposed patch issue reproduced with hs_err (file in attachement):
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # Internal Error 
> (/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/vmThread.cpp:607), pid=26188, 
> tid=26325
> # assert(!t->have_threads_list()) failed: Deadlock if we have exiting threads 
> and if vm thread is running an VM op.
> #
> # JRE version: Java(TM) SE Runtime Environment (12.0) (slowdebug build 
> 12-internal+0-2018-10-24-2022348.lmesnik.hs-bigapps)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (slowdebug 
> 12-internal+0-2018-10-24-2022348.lmesnik.hs-bigapps, mixed mode, sharing, 
> tiered, compressed oops, g1 gc, linux-amd64)
> # Core dump will be written. Default location: Core dumps may be processed with 
> "/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e %P %I %h" (or dumping to 
> /scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_KitchensinkSampler_java/scratch/0/core.26188) 
> 
> #
> # If you would like to submit a bug report, please visit:
> # http://bugreport.java.com/bugreport/crash.jsp
> #
> 
> --------------- S U M M A R Y ------------
> 
> Command Line: -XX:MaxRAMPercentage=2 -XX:MaxRAMPercentage=50 
> -XX:+CrashOnOutOfMemoryError -Djava.net.preferIPv6Addresses=false 
> -XX:-PrintVMOptions -XX:+DisplayVMOutputToStderr -XX:+UsePerfData 
> -Xlog:gc*,gc+heap=debug:gc.log:uptime,timemillis,level,tags 
> -XX:+DisableExplicitGC -XX:+PrintFlagsFinal -XX:+StartAttachListener 
> -XX:NativeMemoryTracking=detail -XX:+FlightRecorder 
> --add-exports=java.base/java.lang=ALL-UNNAMED 
> --add-opens=java.base/java.lang=ALL-UNNAMED 
> --add-exports=java.xml/com.sun.org.apache.xerces.internal.parsers=ALL-UNNAMED 
> --add-exports=java.xml/com.sun.org.apache.xerces.internal.util=ALL-UNNAMED 
> -Djava.io.tmpdir=/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_KitchensinkSampler_java/scratch/0/java.io.tmpdir 
> -Duser.home=/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_KitchensinkSampler_java/scratch/0/user.home 
> -agentpath:/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/images/test/hotspot/jtreg/native/libJvmtiStressModule.so 
> applications.kitchensink.process.stress.Main 
> /scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_KitchensinkSampler_java/scratch/0/kitchensink.final.properties 
> 
> 
> Host: Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz, 32 cores, 235G, Oracle Linux 
> Server release 7.3
> Time: Wed Oct 24 13:28:30 2018 PDT elapsed time: 3 seconds (0d 0h 0m 3s)
> 
> --------------- T H R E A D ---------------
> 
> Current thread (0x00002b9f68006000): JavaThread "Jvmti-AgentSampler" daemon 
> [_thread_in_vm, id=26325, stack(0x00002b9f88808000,0x00002b9f88909000)] 
> _threads_hazard_ptr=0x00002b9f68008e30, _nested_threads_hazard_ptr_cnt=0
> 
> Stack: [0x00002b9f88808000,0x00002b9f88909000], sp=0x00002b9f88907440, free 
> space=1021k
> Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, 
> Vv=VM code, C=native code)
> V [libjvm.so+0x12a04bb] VMError::report_and_die(int, char const*, char const*, 
> __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, 
> unsigned long)+0x6a5
> V [libjvm.so+0x129fdb3] VMError::report_and_die(Thread*, void*, char const*, 
> int, char const*, char const*, __va_list_tag*)+0x57
> V [libjvm.so+0x8ca5ab] report_vm_error(char const*, int, char const*, char 
> const*, ...)+0x152
> V [libjvm.so+0x12e485b] VMThread::execute(VM_Operation*)+0x99
> V [libjvm.so+0xdbec54] JvmtiEnv::GetStackTrace(JavaThread*, int, int, 
> _jvmtiFrameInfo*, int*)+0xc0
> V [libjvm.so+0xd677cf] jvmti_GetStackTrace+0x2c2
> C [libJvmtiStressModule.so+0x302d] trace_stack+0xa9
> C [libJvmtiStressModule.so+0x3daf] agent_sampler+0x21f
> V [libjvm.so+0xddf595] JvmtiAgentThread::call_start_function()+0x67
> V [libjvm.so+0xddf52a] JvmtiAgentThread::start_function_wrapper(JavaThread*, 
> Thread*)+0xf2
> V [libjvm.so+0x1218945] JavaThread::thread_main_inner()+0x17f
> V [libjvm.so+0x12187ad] JavaThread::run()+0x273
> V [libjvm.so+0x100e4ee] thread_native_entry(Thread*)+0x192
> 
> 
> 
> Leonid attached the full hs_err log to the bug report.
> 
> Thanks,
> Serguei
> 
> 
> On 10/24/18 02:00, serguei.spitsyn at oracle.com wrote:
>> Hi Robbin and David,
>>
>> There is no JvmtiEnv::SuspendThreadList call in the dumped stack traces.
>> But there is an instance of the JvmtiEnv::SuspendThread which seems to be 
>> supporting your theory:
>>
>> Thread 136 (Thread 0x2ae494100700 (LWP 28023)):
>> #0  0x00002ae3927b5945 in pthread_cond_wait@@GLIBC_2.3.2 () from 
>> /lib64/libpthread.so.0
>> #1  0x00002ae393ba8d63 in os::PlatformEvent::park 
>> (this=this at entry=0x2ae454005800) at 
>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897
>> #2  0x00002ae393b50cf8 in ParkCommon (timo=0, ev=0x2ae454005800) at 
>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399
>> #3  Monitor::IWait (this=this at entry=0x2ae398023c10, 
>> Self=Self at entry=0x2ae454004800, timo=timo at entry=0) at 
>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:768
>> #4  0x00002ae393b51f2e in Monitor::wait (this=this at entry=0x2ae398023c10, 
>> no_safepoint_check=<optimized out>, timeout=timeout at entry=0, 
>> as_suspend_equivalent=as_suspend_equivalent at entry=false) at 
>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:1106
>> #5  0x00002ae393de7867 in VMThread::execute (op=op at entry=0x2ae4940ffb10) at 
>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/vmThread.cpp:657
>> #6  0x00002ae393d6a3bd in JavaThread::java_suspend 
>> (this=this at entry=0x2ae3985f2000) at 
>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:2321
>> #7  0x00002ae3939ad7e1 in JvmtiSuspendControl::suspend 
>> (java_thread=java_thread at entry=0x2ae3985f2000) at 
>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiImpl.cpp:847
>> #8  0x00002ae3939887ae in JvmtiEnv::SuspendThread 
>> (this=this at entry=0x2ae39801b270, java_thread=0x2ae3985f2000) at 
>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEnv.cpp:955
>> #9  0x00002ae39393a8c6 in jvmti_SuspendThread (env=0x2ae39801b270, 
>> thread=0x2ae49929fdf8) at 
>> /scratch/lmesnik/ws/hs-bigapps/build/linux-x64/hotspot/variant-server/gensrc/jvmtifiles/jvmtiEnter.cpp:527 
>>
>> #10 0x00002ae394d973ee in agent_sampler (jvmti=0x2ae39801b270, env=<optimized 
>> out>, p=<optimized out>) at 
>> /scratch/lmesnik/ws/hs-bigapps/closed/test/hotspot/jtreg/applications/kitchensink/process/stress/modules/libJvmtiStressModule.c:274 
>>
>> #11 0x00002ae3939ab24d in call_start_function (this=0x2ae454004800) at 
>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiImpl.cpp:85
>> #12 JvmtiAgentThread::start_function_wrapper (thread=0x2ae454004800, 
>> __the_thread__=<optimized out>) at 
>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiImpl.cpp:79
>> #13 0x00002ae393d7338a in JavaThread::thread_main_inner 
>> (this=this at entry=0x2ae454004800) at 
>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1795
>> #14 0x00002ae393d736c6 in JavaThread::run (this=0x2ae454004800) at 
>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1775
>> #15 0x00002ae393ba0070 in thread_native_entry (thread=0x2ae454004800) at 
>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/linux/os_linux.cpp:698
>> #16 0x00002ae3927b1e25 in start_thread () from /lib64/libpthread.so.0
>> #17 0x00002ae392cc234d in clone () from /lib64/libc.so.6
>>
>>
>> The JavaThread::suspend() call from the stack trace above also holds a 
>> ThreadsListHandle while executing VMThread::execute(&vm_suspend):
>>
>> void JavaThread::java_suspend() {
>>   ThreadsListHandle tlh;
>>   if (!tlh.includes(this) || threadObj() == NULL || is_exiting()) {
>>     return;
>>   }
>>
>>   { MutexLockerEx ml(SR_lock(), Mutex::_no_safepoint_check_flag);
>>     if (!is_external_suspend()) {
>>       // a racing resume has cancelled us; bail out now
>>       return;
>>     }
>>
>>     // suspend is done
>>     uint32_t debug_bits = 0;
>>     // Warning: is_ext_suspend_completed() may temporarily drop the
>>     // SR_lock to allow the thread to reach a stable thread state if
>>     // it is currently in a transient thread state.
>>     if (is_ext_suspend_completed(false /* !called_by_wait */,
>>                                  SuspendRetryDelay, &debug_bits)) {
>>       return;
>>     }
>>   }
>>
>>   VM_ThreadSuspend vm_suspend;
>>   VMThread::execute(&vm_suspend);
>> }
>>
>>
>> I'll check with Leonid tomorrow if we can check if your patch below can fix 
>> this deadlock.
>>
>> Thanks,
>> Serguei
>>
>>
>> On 10/24/18 00:18, Robbin Ehn wrote:
>>> Hi, truncate:
>>>
>>> On 24/10/2018 02:00, serguei.spitsyn at oracle.com wrote:
>>>>>> One thing I noticed which Robbin should be able to expand upon is that 
>>>>>> Thread 101 is terminating and has called ThreadsSMRSupport::smr_delete and 
>>>>>> is blocked here:
>>>>>>
>>>>>>   // Wait for a release_stable_list() call before we check again. No
>>>>>>   // safepoint check, no timeout, and not as suspend equivalent flag
>>>>>>   // because this JavaThread is not on the Threads list.
>>>>>>   ThreadsSMRSupport::delete_lock()->wait(Mutex::_no_safepoint_check_flag,
>>>>>>                                          0, 
>>>>>> !Mutex::_as_suspend_equivalent_flag);
>>>>>>
>>>>>> As the comment says this thread is no longer on the Threads_list, but the 
>>>>>> VM_HandshakeAllThreads is not a safepoint operation and does not hold the 
>>>>>> Threads_lock, so is it possible this thread was captured by the 
>>>>>> JavaThreadIteratorWithHandle being used by VM_HandshakeAllThreads, before 
>>>>>> it got removed? If so we'd be hung waiting it for it handshake as it's not 
>>>>>> in a "safepoint-safe" or suspend-equivalent state.
>>>
>>> In short:
>>> # VM Thread
>>> VM Thread is in a loop, takes Threads_lock, takes a new snapshot of the 
>>> Threads_list, scans the list and process handshakes on behalf of safe threads.
>>> Releases snapshot and Threads_lock and checks if all handshakes are completed
>>>
>>> # An exiting thread
>>> A thread exiting thread removes it self from _THE_ threads list, but must 
>>> stick around if it is on any snapshots of alive. When it is no on any list it 
>>> will cancel the handshake.
>>>
>>> Since VM thread during the handshake takes a new snapshot every iteration any 
>>> exiting can proceed since it will not be a the new snapshot. Thus cancel the 
>>> handshake and VM thread can exit the loop (if this was the last handshake).
>>>
>>> Constraint:
>>> If any thread grabs a snapshot of threads list and later tries to take a lock 
>>> which is 'used' by VM Thread or inside the handshake we can deadlock.
>>>
>>> Considering that looking at e.g. : JvmtiEnv::SuspendThreadList
>>> Which calls VMThread::execute(&tsj); with a ThreadsListHandle alive, this 
>>> could deadlock AFAICT. Since the thread will rest on VMOperationRequest_lock 
>>> with a Threads list snapshot but VM thread cannot finishes handshake until 
>>> that snapshot is released.
>>>
>>> I suggest first step is to add something like this patch below and fix the 
>>> obvious ones first.
>>>
>>> Note, I have not verified that is the problem you are seeing, I'm saying that 
>>> this seem to be real issue. And considering how the stack traces looks, it 
>>> may be this.
>>>
>>> You want me going through this, just assign a bug if there is one?
>>>
>>> /Robbin
>>>
>>> diff -r 622fd3608374 src/hotspot/share/runtime/thread.hpp
>>> --- a/src/hotspot/share/runtime/thread.hpp    Tue Oct 23 13:27:41 2018 +0200
>>> +++ b/src/hotspot/share/runtime/thread.hpp    Wed Oct 24 09:13:17 2018 +0200
>>> @@ -167,2 +167,6 @@
>>>    }
>>> + public:
>>> +  bool have_threads_list();
>>> + private:
>>> +
>>>    // This field is enabled via -XX:+EnableThreadSMRStatistics:
>>> diff -r 622fd3608374 src/hotspot/share/runtime/thread.inline.hpp
>>> --- a/src/hotspot/share/runtime/thread.inline.hpp    Tue Oct 23 13:27:41 2018 
>>> +0200
>>> +++ b/src/hotspot/share/runtime/thread.inline.hpp    Wed Oct 24 09:13:17 2018 
>>> +0200
>>> @@ -111,2 +111,6 @@
>>>
>>> +inline bool Thread::have_threads_list() {
>>> +  return OrderAccess::load_acquire(&_threads_hazard_ptr) != NULL;
>>> +}
>>> +
>>>  inline void Thread::set_threads_hazard_ptr(ThreadsList* new_list) {
>>> diff -r 622fd3608374 src/hotspot/share/runtime/vmThread.cpp
>>> --- a/src/hotspot/share/runtime/vmThread.cpp    Tue Oct 23 13:27:41 2018 +0200
>>> +++ b/src/hotspot/share/runtime/vmThread.cpp    Wed Oct 24 09:13:17 2018 +0200
>>> @@ -608,2 +608,3 @@
>>>    if (!t->is_VM_thread()) {
>>> +    assert(t->have_threads_list(), "Deadlock if we have exiting threads and 
>>> if vm thread is running an VM op."); // fatal/guarantee
>>>      SkipGCALot sgcalot(t);    // avoid re-entrant attempts to gc-a-lot
>>>
>>>
>>>
>>>
>>>
>>>>>>
>>>>>> David
>>>>>> -----
>>>>>>
>>>>>> On 24/10/2018 9:18 AM, serguei.spitsyn at oracle.com wrote:
>>>>>>> Please, skip it - sorry for the noise.
>>>>>>> It is hard to prove anything with current dump.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Serguei
>>>>>>>
>>>>>>> On 10/23/18 9:09 AM, serguei.spitsyn at oracle.com wrote:
>>>>>>>> Hi David and Robbin,
>>>>>>>>
>>>>>>>> I have an idea that needs to be checked.
>>>>>>>> It can be almost the same deadlock scenario that I've already explained 
>>>>>>>> but more sophisticated.
>>>>>>>> I suspect a scenario with JvmtiThreadState_lock that the flag 
>>>>>>>> Monitor::_safepoint_check_always does not help much.
>>>>>>>> It can be verified by checking what monitors are used by the blocked 
>>>>>>>> threads.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Serguei
>>>>>>>>
>>>>>>>>
>>>>>>>> On 10/23/18 07:38, Robbin Ehn wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> On 10/23/18 10:34 AM, David Holmes wrote:
>>>>>>>>>> Hi Serguei,
>>>>>>>>>>
>>>>>>>>>> The VMThread is executing VM_HandshakeAllThreads which is not a 
>>>>>>>>>> safepoint operation. There's no real way to tell from the stacks what 
>>>>>>>>>> it's stuck on.
>>>>>>>>>
>>>>>>>>> I cannot find a thread that is not considered safepoint safe or 
>>>>>>>>> is_ext_suspended (thread 146). So the handshake should go through. The 
>>>>>>>>> handshake will log a warning after a while, is there such warning from 
>>>>>>>>> the handshake operation?
>>>>>>>>>
>>>>>>>>> There are several threads competing with e.g. Threads_lock, and threads 
>>>>>>>>> waiting for GC and several other VM ops, could it just be really slow?
>>>>>>>>>
>>>>>>>>> /Robbin
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> David
>>>>>>>>>>
>>>>>>>>>> On 23/10/2018 5:58 PM, serguei.spitsyn at oracle.com wrote:
>>>>>>>>>>> Hi David,
>>>>>>>>>>>
>>>>>>>>>>> You are right, thanks.
>>>>>>>>>>> It means, this deadlock needs more analysis.
>>>>>>>>>>> For completeness, the stack traces are in attachments.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Serguei
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 10/23/18 00:43, David Holmes wrote:
>>>>>>>>>>>> Hi Serguei,
>>>>>>>>>>>>
>>>>>>>>>>>> The JvmtiThreadState_lock is always acquired with safepoint checks 
>>>>>>>>>>>> enabled, so all JavaThreads blocked trying to acquire it will be 
>>>>>>>>>>>> _thread_blocked and so safepoint-safe and so won't be holding up the 
>>>>>>>>>>>> safepoint.
>>>>>>>>>>>>
>>>>>>>>>>>> David
>>>>>>>>>>>>
>>>>>>>>>>>> On 23/10/2018 5:21 PM, serguei.spitsyn at oracle.com wrote:
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've added the seviceability-dev mailing list.
>>>>>>>>>>>>> It can be interesting for the SVC folks. :)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 10/22/18 22:14, Leonid Mesnik wrote:
>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Seems last version also crashes with 2 other different symptoms.
>>>>>>>>>>>>>> http://java.se.oracle.com:10065/mdash/jobs/lmesnik-ks8-20181021-0638-7157/results?search=status%3Afailed+AND+-state%3Ainvalid 
>>>>>>>>>>>>>> <http://java.se.oracle.com:10065/mdash/jobs/lmesnik-ks8-20181021-0638-7157/results?search=status:failed+AND+-state:invalid> 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also it might hangs with  stack attached.  Seems that test might 
>>>>>>>>>>>>>> be blocked because it invoke 2 jvmti methods. Can jvmti agent 
>>>>>>>>>>>>>> invoke jvmti methods from different threads?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes, in general.
>>>>>>>>>>>>> However, you have to be careful when using debugging features.
>>>>>>>>>>>>> Below, one thread is enabling single stepping while another thread 
>>>>>>>>>>>>> is being suspended.
>>>>>>>>>>>>> Both are blocked at a safepoint which is Okay in general but not 
>>>>>>>>>>>>> Okay if they hold any lock.
>>>>>>>>>>>>> For instance, the thread #152 is holding the monitor JvmtiThreadState.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Also, I see a couple of more threads that are interesting as well:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thread 159 (Thread 0x2ae40b78f700 (LWP 27962)):
>>>>>>>>>>>>> #0  0x00002ae3927b5945 in pthread_cond_wait@@GLIBC_2.3.2 () from 
>>>>>>>>>>>>> /lib64/libpthread.so.0
>>>>>>>>>>>>> #1  0x00002ae393ba8d63 in os::PlatformEvent::park 
>>>>>>>>>>>>> (this=this at entry=0x2ae3984c9100) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #2  0x00002ae393b50920 in ParkCommon (timo=0, ev=0x2ae3984c9100) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #3  Monitor::ILock (this=0x2ae398024f10, Self=0x2ae3984c7800) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:461 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #4  0x00002ae393b512c1 in lock 
>>>>>>>>>>>>> (Self=0x2ae3984c7is_ext_suspended800, this=0x2ae398024f10) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:910 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #5  Monitor::lock (this=this at entry=0x2ae398024f10) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:919 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #6  0x00002ae39350510c in MutexLocker (mutex=0x2ae398024f10, 
>>>>>>>>>>>>> this=<synthetic pointer>) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutexLocker.hpp:182 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #7  ciEnv::cache_jvmti_state (this=this at entry=0x2ae40b78eb30) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/ci/ciEnv.cpp:229
>>>>>>>>>>>>> #8  0x00002ae3935d3294 in CompileBroker::invoke_compiler_on_method 
>>>>>>>>>>>>> (task=task at entry=0x2ae48800ff40) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/compiler/compileBroker.cpp:2084 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #9  0x00002ae3935d4f48 in CompileBroker::compiler_thread_loop () at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/compiler/compileBroker.cpp:1798 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #10 0x00002ae393d7338a in JavaThread::thread_main_inner 
>>>>>>>>>>>>> (this=this at entry=0x2ae3984c7800) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1795 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #11 0x00002ae393d736c6 in JavaThread::run (this=0x2ae3984c7800) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1775 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #12 0x00002ae393ba0070 in thread_native_entry 
>>>>>>>>>>>>> (thread=0x2ae3984c7800) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/linux/os_linux.cpp:698 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #13 0x00002ae3927b1e25 in start_thread () from /lib64/libpthread.so.0
>>>>>>>>>>>>> #14 0x00002ae392cc234d in clone () from /lib64/libc.so.6
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thread 158 (Thread 0x2ae40b890700 (LWP 27963)):
>>>>>>>>>>>>> #0  0x00002ae3927b5945 in pthread_cond_wait@@GLIBC_2.3.2 () from 
>>>>>>>>>>>>> /lib64/libpthread.so.0
>>>>>>>>>>>>> #1  0x00002ae393ba8d63 in os::PlatformEvent::park 
>>>>>>>>>>>>> (this=this at entry=0x2ae3984cbb00) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #2  0x00002ae393b50920 in ParkCommon (timo=0, ev=0x2ae3984cbb00) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #3  Monitor::ILock (this=0x2ae398024f10, Self=0x2ae3984ca800) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:461 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #4  0x00002ae393b512c1 in lock (Self=0x2ae3984ca800, 
>>>>>>>>>>>>> this=0x2ae398024f10) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:910 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #5  Monitor::lock (this=this at entry=0x2ae398024f10) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:919 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #6  0x00002ae39350510c in MutexLocker (mutex=0x2ae398024f10, 
>>>>>>>>>>>>> this=<synthetic pointer>) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutexLocker.hpp:182 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #7  ciEnv::cache_jvmti_state (this=this at entry=0x2ae40b88fb30) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/ci/ciEnv.cpp:229
>>>>>>>>>>>>> #8  0x00002ae3935d3294 in CompileBroker::invoke_compiler_on_method 
>>>>>>>>>>>>> (task=task at entry=0x2ae49c00a670) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/compiler/compileBroker.cpp:2084 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #9  0x00002ae3935d4f48 in CompileBroker::compiler_thread_loop () at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/compiler/compileBroker.cpp:1798 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #10 0x00002ae393d7338a in JavaThread::thread_main_inner 
>>>>>>>>>>>>> (this=this at entry=0x2ae3984ca800) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1795 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #11 0x00002ae393d736c6 in JavaThread::run (this=0x2ae3984ca800) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1775 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #12 0x00002ae393ba0070 in thread_native_entry 
>>>>>>>>>>>>> (thread=0x2ae3984ca800) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/linux/os_linux.cpp:698 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #13 0x00002ae3927b1e25 in start_thread () from /lib64/libpthread.so.0
>>>>>>>>>>>>> #14 0x00002ae392cc234d in clone () from /lib64/libc.so.6
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thread 51 (Thread 0x2ae49549b700 (LWP 29678)):
>>>>>>>>>>>>> #0  0x00002ae3927b5945 in pthread_cond_wait@@GLIBC_2.3.2 () from 
>>>>>>>>>>>>> /lib64/libpthread.so.0
>>>>>>>>>>>>> #1  0x00002ae393ba8d63 in os::PlatformEvent::park 
>>>>>>>>>>>>> (this=this at entry=0x2ae460061c00) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #2  0x00002ae393b50920 in ParkCommon (timo=0, ev=0x2ae460061c00) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #3  Monitor::ILock (this=0x2ae398024f10, Self=0x2ae4600c2800) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:461 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #4  0x00002ae393b512c1 in lock (Self=0x2ae4600c2800, 
>>>>>>>>>>>>> this=0x2ae398024f10) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:910 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #5  Monitor::lock (this=this at entry=0x2ae398024f10) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:919 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #6  0x00002ae393999682 in MutexLocker (mutex=0x2ae398024f10, 
>>>>>>>>>>>>> this=<synthetic pointer>) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutexLocker.hpp:182 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #7  thread_started (thread=0x2ae4600c2800) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEventController.cpp:668 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #8  JvmtiEventController::thread_started (thread=0x2ae4600c2800) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEventController.cpp:1027 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #9  0x00002ae39399f3a0 in JvmtiExport::post_thread_start 
>>>>>>>>>>>>> (thread=thread at entry=0x2ae4600c2800) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiExport.cpp:1395 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #10 0x00002ae393d737d8 in JavaThread::run (this=0x2ae4600c2800) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1764 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #11 0x00002ae393ba0070 in thread_native_entry 
>>>>>>>>>>>>> (thread=0x2ae4600c2800) at 
>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/linux/os_linux.cpp:698 
>>>>>>>>>>>>>
>>>>>>>>>>>>> #12 0x00002ae3927b1e25 in start_thread () from /lib64/libpthread.so.0
>>>>>>>>>>>>> #13 0x00002ae392cc234d in clone () from /lib64/libc.so.6
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> These two thread are blocked on the monitor JvmtiThreadState_lock 
>>>>>>>>>>>>> in the function ciEnv::cache_jvmti_state().
>>>>>>>>>>>>> Also, there are many threads (like #51) that are executing 
>>>>>>>>>>>>> JvmtiExport::post_thread_start and blocked on the same monitor.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Now, the question is why this safepoint can not start?
>>>>>>>>>>>>> What thread is blocking it? Or in reverse, what thread this 
>>>>>>>>>>>>> safepoint is waiting for?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think, this safepoint operation is waiting for all threads that 
>>>>>>>>>>>>> are blocked on the JvmtiThreadState_lock.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Conclusion:
>>>>>>>>>>>>>
>>>>>>>>>>>>> The deadlock is:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thread #152:
>>>>>>>>>>>>>    - grabbed the monitor JvmtiThreadState_lock
>>>>>>>>>>>>>    - blocked in the VM_GetCurrentLocation in the function 
>>>>>>>>>>>>> JvmtiEnvThreadState::reset_current_location()
>>>>>>>>>>>>>
>>>>>>>>>>>>> Many other threads:
>>>>>>>>>>>>>    - blocked on the monitor JvmtiThreadState_lock
>>>>>>>>>>>>>    - can not reach the blocked at a safepoint state (all threads 
>>>>>>>>>>>>> have to reach this state for this safepoint to happen)
>>>>>>>>>>>>>
>>>>>>>>>>>>> It seems to me, this is a bug which has to be filed.
>>>>>>>>>>>>>
>>>>>>>>>>>>> My guess is that this will stop to reproduce after if you turn off 
>>>>>>>>>>>>> the single stepping for thread #152.
>>>>>>>>>>>>> Please, let me know about the results.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Assuming that crashes look like VM bugs I think it make sense to 
>>>>>>>>>>>>>> integrate jvmti changes but *don't* enabled jvmti module by default.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This one is a deadlock.
>>>>>>>>>>>>> However, the root cause is a race condition that can potentially 
>>>>>>>>>>>>> result in both deadlocks and crashes.
>>>>>>>>>>>>> So, I'm curious if you observed crashes as well.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> And add to more tests with jvmti enabled.
>>>>>>>>>>>>>> So anyone could easily run them to reproduce crashes.  This test 
>>>>>>>>>>>>>> would be out of CI to don't introduce any bugs. Does it make sense?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Consider hang - I think that it might be product bug since I don't 
>>>>>>>>>>>>>> see any locking on my monitors. But I am not sure. Is it possible 
>>>>>>>>>>>>>> that any my code jvmti agent prevent VM to get into safepoint?
>>>>>>>>>>>>>> Could we discuss it tomorrow or his week when you have a time?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes, of course.
>>>>>>>>>>>>> Let's find some time tomorrow.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Any suggestion how to diagnose deadlock would be great.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Analysis of stack traces is needed.
>>>>>>>>>>>>> It is non-trivial in this particular case as there are so many 
>>>>>>>>>>>>> threads executed at the same time.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Part of stack trace with 2 my threads only:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thread 136 (Thread 0x2ae494100700 (LWP 28023)):
>>>>>>>>>>>>>> #0  0x00002ae3927b5945 in pthread_cond_wait@@GLIBC_2.3.2 () from 
>>>>>>>>>>>>>> /lib64/libpthread.so.0
>>>>>>>>>>>>>> #1  0x00002ae393ba8d63 in os::PlatformEvent::park 
>>>>>>>>>>>>>> (this=this at entry=0x2ae454005800) at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> #2  0x00002ae393b50cf8 in ParkCommon (timo=0, ev=0x2ae454005800) 
>>>>>>>>>>>>>> at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> #3  Monitor::IWait (this=this at entry=0x2ae398023c10, 
>>>>>>>>>>>>>> Self=Self at entry=0x2ae454004800, timo=timo at entry=0) at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:76\ 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 8
>>>>>>>>>>>>>> #4  0x00002ae393b51f2e in Monitor::wait 
>>>>>>>>>>>>>> (this=this at entry=0x2ae398023c10, no_safepoint_check=<optimized 
>>>>>>>>>>>>>> out>, timeout=timeout at entry=0, 
>>>>>>>>>>>>>> as_suspend_equivalent=as_suspend_equivalent at en\
>>>>>>>>>>>>>> try=false) at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:1106 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> #5  0x00002ae393de7867 in VMThread::execute 
>>>>>>>>>>>>>> (op=op at entry=0x2ae4940ffb10) at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/vmThread.cpp:657 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> #6  0x00002ae393d6a3bd in JavaThread::java_suspend 
>>>>>>>>>>>>>> (this=this at entry=0x2ae3985f2000) at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:2321 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> #7  0x00002ae3939ad7e1 in JvmtiSuspendControl::suspend 
>>>>>>>>>>>>>> (java_thread=java_thread at entry=0x2ae3985f2000) at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiImpl.cpp:8\ 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 47
>>>>>>>>>>>>>> #8  0x00002ae3939887ae in JvmtiEnv::SuspendThread 
>>>>>>>>>>>>>> (this=this at entry=0x2ae39801b270, java_thread=0x2ae3985f2000) at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiE\
>>>>>>>>>>>>>> nv.cpp:955
>>>>>>>>>>>>>> #9  0x00002ae39393a8c6 in jvmti_SuspendThread (env=0x2ae39801b270, 
>>>>>>>>>>>>>> thread=0x2ae49929fdf8) at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/build/linux-x64/hotspot/variant-server/gensrc/jvmtifiles\ 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> /jvmtiEnter.cpp:527
>>>>>>>>>>>>>> #10 0x00002ae394d973ee in agent_sampler (jvmti=0x2ae39801b270, 
>>>>>>>>>>>>>> env=<optimized out>, p=<optimized out>) at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/closed/test/hotspot/jtreg/applications/kitc\ 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> hensink/process/stress/modules/libJvmtiStressModule.c:274
>>>>>>>>>>>>>> #11 0x00002ae3939ab24d in call_start_function 
>>>>>>>>>>>>>> (this=0x2ae454004800) at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiImpl.cpp:85 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> #12 JvmtiAgentThread::start_function_wrapper 
>>>>>>>>>>>>>> (thread=0x2ae454004800, __the_thread__=<optimized out>) at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiImpl.cpp:79 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> #13 0x00002ae393d7338a in JavaThread::thread_main_inner 
>>>>>>>>>>>>>> (this=this at entry=0x2ae454004800) at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1795 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> #14 0x00002ae393d736c6 in JavaThread::run (this=0x2ae454004800) at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1775 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> #15 0x00002ae393ba0070 in thread_native_entry 
>>>>>>>>>>>>>> (thread=0x2ae454004800) at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/linux/os_linux.cpp:698 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> #16 0x00002ae3927b1e25 in start_thread () from /lib64/libpthread.so.0
>>>>>>>>>>>>>> #17 0x00002ae392cc234d in clone () from /lib64/libc.so.6
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thread 152 (Thread 0x2ae427060700 (LWP 27995)):
>>>>>>>>>>>>>> #0  0x00002ae3927b5945 in pthread_cond_wait@@GLIBC_2.3.2 () from 
>>>>>>>>>>>>>> /lib64/libpthread.so.0
>>>>>>>>>>>>>> #1  0x00002ae393ba8d63 in os::PlatformEvent::park 
>>>>>>>>>>>>>> (this=this at entry=0x2ae3985e7400) at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> #2  0x00002ae393b50cf8 in ParkCommon (timo=0, ev=0x2ae3985e7400) 
>>>>>>>>>>>>>> at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> #3  Monitor::IWait (this=this at entry=0x2ae398023c10, 
>>>>>>>>>>>>>> Self=Self at entry=0x2ae3985e6000, timo=timo at entry=0) at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:76\ 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 8
>>>>>>>>>>>>>> #4  0x00002ae393b51f2e in Monitor::wait 
>>>>>>>>>>>>>> (this=this at entry=0x2ae398023c10, no_safepoint_check=<optimized 
>>>>>>>>>>>>>> out>, timeout=timeout at entry=0, 
>>>>>>>>>>>>>> as_suspend_equivalent=as_suspend_equivalent at en\
>>>>>>>>>>>>>> try=false) at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:1106 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> #5  0x00002ae393de7867 in VMThread::execute (op=0x2ae42705f500) at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/vmThread.cpp:657 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> #6  0x00002ae3939965f3 in 
>>>>>>>>>>>>>> JvmtiEnvThreadState::reset_current_location 
>>>>>>>>>>>>>> (this=this at entry=0x2ae6bc000d80, 
>>>>>>>>>>>>>> event_type=event_type at entry=JVMTI_EVENT_SINGLE_STEP, 
>>>>>>>>>>>>>> enabled=enabled at entry=tr\
>>>>>>>>>>>>>> ue) at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEnvThreadState.cpp:312 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> #7  0x00002ae393997acf in recompute_env_thread_enabled 
>>>>>>>>>>>>>> (state=0x2ae6bc000cd0, ets=0x2ae6bc000d80) at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEventControlle\ 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> r.cpp:490
>>>>>>>>>>>>>> #8  JvmtiEventControllerPrivate::recompute_thread_enabled 
>>>>>>>>>>>>>> (state=state at entry=0x2ae6bc000cd0) at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEventController.cpp\ 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> :523
>>>>>>>>>>>>>> #9  0x00002ae393998168 in 
>>>>>>>>>>>>>> JvmtiEventControllerPrivate::recompute_enabled () at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEventController.cpp:598 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> #10 0x00002ae39399a244 in set_user_enabled (enabled=true, 
>>>>>>>>>>>>>> event_type=JVMTI_EVENT_SINGLE_STEP, thread=0x0, 
>>>>>>>>>>>>>> env=0x2ae39801b270) at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/sha\
>>>>>>>>>>>>>> re/prims/jvmtiEventController.cpp:818
>>>>>>>>>>>>>> #11 JvmtiEventController::set_user_enabled (env=0x2ae39801b270, 
>>>>>>>>>>>>>> thread=0x0, event_type=JVMTI_EVENT_SINGLE_STEP, enabled=<optimized 
>>>>>>>>>>>>>> out>) at /scratch/lmesnik/ws/hs-bigapps/open/src/\
>>>>>>>>>>>>>> hotspot/share/prims/jvmtiEventController.cpp:963
>>>>>>>>>>>>>> #12 0x00002ae393987d2d in JvmtiEnv::SetEventNotificationMode 
>>>>>>>>>>>>>> (this=this at entry=0x2ae39801b270, mode=mode at entry=JVMTI_ENABLE, 
>>>>>>>>>>>>>> event_type=event_type at entry=JVMTI_EVENT_SINGLE_STEP, eve\
>>>>>>>>>>>>>> nt_thread=event_thread at entry=0x0) at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEnv.cpp:543 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> #13 0x00002ae3939414eb in jvmti_SetEventNotificationMode 
>>>>>>>>>>>>>> (env=0x2ae39801b270, mode=mode at entry=JVMTI_ENABLE, 
>>>>>>>>>>>>>> event_type=event_type at entry=JVMTI_EVENT_SINGLE_STEP, 
>>>>>>>>>>>>>> event_thread=event_\
>>>>>>>>>>>>>> thread at entry=0x0) at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/build/linux-x64/hotspot/variant-server/gensrc/jvmtifiles/jvmtiEnter.cpp:5389 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> #14 0x00002ae394d97989 in enable_events () at 
>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/closed/test/hotspot/jtreg/applications/kitchensink/process/stress/modules/libJvmtiStressModule.c:519 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> #15 0x00002ae394d98070 in 
>>>>>>>>>>>>>> Java_applications_kitchensink_process_stress_modules_JvmtiStressModule_startIteration 
>>>>>>>>>>>>>> (env=<optimized out>, this=<optimized out>) at /scratch/lmesnik/ws/h\
>>>>>>>>>>>>>> s-bigapps/closed/test/hotspot/jtreg/applications/kitchensink/process/stress/modules/libJvmtiStressModule.c:697 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> #16 0x00002ae3a43ef257 in ?? ()
>>>>>>>>>>>>>> #17 0x00002ae3a43eede1 in ?? ()
>>>>>>>>>>>>>> #18 0x00002ae42705f878 in ?? ()
>>>>>>>>>>>>>> #19 0x00002ae40ad334e0 in ?? ()
>>>>>>>>>>>>>> #20 0x00002ae42705f8e0 in ?? ()
>>>>>>>>>>>>>> #21 0x00002ae40ad33c68 in ?? ()
>>>>>>>>>>>>>> #22 0x0000000000000000 in ?? ()
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Serguei
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Leonid
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Oct 9, 2018, at 4:52 PM, serguei.spitsyn at oracle.com 
>>>>>>>>>>>>>>> <mailto:serguei.spitsyn at oracle.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Leonid,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> There is an existing bug:
>>>>>>>>>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8043571
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Serguei
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 10/9/18 16:11, Leonid Mesnik wrote:
>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> During fixing kitchensink I get
>>>>>>>>>>>>>>>> assert(_cur_stack_depth == count_frames()) failed: 
>>>>>>>>>>>>>>>> cur_stack_depth out of sync
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Do you know if i might be bug in my jvmti agent?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Leonid
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> #
>>>>>>>>>>>>>>>> # A fatal error has been detected by the Java Runtime Environment:
>>>>>>>>>>>>>>>> #
>>>>>>>>>>>>>>>> #  Internal Error 
>>>>>>>>>>>>>>>> (/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiThreadState.cpp:277), 
>>>>>>>>>>>>>>>> pid=13926, tid=13962
>>>>>>>>>>>>>>>> #  assert(_cur_stack_depth == count_frames()) failed: 
>>>>>>>>>>>>>>>> cur_stack_depth out of sync
>>>>>>>>>>>>>>>> #
>>>>>>>>>>>>>>>> # JRE version: Java(TM) SE Runtime Environment (12.0) (fastdebug 
>>>>>>>>>>>>>>>> build 12-internal+0-2018-10-08-2342517.lmesnik.hs-bigapps)
>>>>>>>>>>>>>>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 
>>>>>>>>>>>>>>>> 12-internal+0-2018-10-08-2342517.lmesnik.hs-bigapps, mixed mode, 
>>>>>>>>>>>>>>>> tiered, compressed oops, g1 gc, linux-amd64)
>>>>>>>>>>>>>>>> # Core dump will be written. Default location: Core dumps may be 
>>>>>>>>>>>>>>>> processed with "/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e 
>>>>>>>>>>>>>>>> %P %I %h" (or dumping to 
>>>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_KitchensinkSanity_java/scratch/0/core.13926) 
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> #
>>>>>>>>>>>>>>>> # If you would like to submit a bug report, please visit:
>>>>>>>>>>>>>>>> # http://bugreport.java.com/bugreport/crash.jsp
>>>>>>>>>>>>>>>> #
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---------------  S U M M A R Y ------------
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Command Line: -XX:MaxRAMPercentage=2 -XX:MaxRAMPercentage=50 
>>>>>>>>>>>>>>>> -XX:+CrashOnOutOfMemoryError 
>>>>>>>>>>>>>>>> -Djava.net.preferIPv6Addresses=false -XX:-PrintVMOptions 
>>>>>>>>>>>>>>>> -XX:+DisplayVMOutputToStderr -XX:+UsePerfData 
>>>>>>>>>>>>>>>> -Xlog:gc*,gc+heap=debug:gc.log:uptime,timemillis,level,tags 
>>>>>>>>>>>>>>>> -XX:+DisableExplicitGC -XX:+PrintFlagsFinal 
>>>>>>>>>>>>>>>> -XX:+StartAttachListener -XX:NativeMemoryTracking=detail 
>>>>>>>>>>>>>>>> -XX:+FlightRecorder 
>>>>>>>>>>>>>>>> --add-exports=java.base/java.lang=ALL-UNNAMED 
>>>>>>>>>>>>>>>> --add-opens=java.base/java.lang=ALL-UNNAMED 
>>>>>>>>>>>>>>>> --add-exports=java.xml/com.sun.org.apache.xerces.internal.parsers=ALL-UNNAMED 
>>>>>>>>>>>>>>>> --add-exports=java.xml/com.sun.org.apache.xerces.internal.util=ALL-UNNAMED 
>>>>>>>>>>>>>>>> -Djava.io.tmpdir=/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_KitchensinkSanity_java/scratch/0/java.io.tmpdir 
>>>>>>>>>>>>>>>> -Duser.home=/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_KitchensinkSanity_java/scratch/0/user.home 
>>>>>>>>>>>>>>>> -agentpath:/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/images/test/hotspot/jtreg/native/libJvmtiStressModule.so 
>>>>>>>>>>>>>>>> applications.kitchensink.process.stress.Main 
>>>>>>>>>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_KitchensinkSanity_java/scratch/0/kitchensink.final.properties 
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Host: scaaa118.us.oracle.com <http://scaaa118.us.oracle.com>, 
>>>>>>>>>>>>>>>> Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz, 32 cores, 235G, Oracle 
>>>>>>>>>>>>>>>> Linux Server release 7.3
>>>>>>>>>>>>>>>> Time: Tue Oct  9 16:06:07 2018 PDT elapsed time: 31 seconds (0d 
>>>>>>>>>>>>>>>> 0h 0m 31s)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---------------  T H R E A D  ---------------
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Current thread (0x00002af3dc6ac800):  VMThread "VM Thread" 
>>>>>>>>>>>>>>>> [stack: 0x00002af44f10a000,0x00002af44f20a000] [id=13962] 
>>>>>>>>>>>>>>>> _threads_hazard_ptr=0x00002af4ac090eb0, 
>>>>>>>>>>>>>>>> _nested_threads_hazard_ptr_cnt=0
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Stack: [0x00002af44f10a000,0x00002af44f20a000], 
>>>>>>>>>>>>>>>>  sp=0x00002af44f208720,  free space=1017k
>>>>>>>>>>>>>>>> Native frames: (J=compiled Java code, A=aot compiled Java code, 
>>>>>>>>>>>>>>>> j=interpreted, Vv=VM code, C=native code)
>>>>>>>>>>>>>>>> V  [libjvm.so+0x18c4923]  VMError::report_and_die(int, char 
>>>>>>>>>>>>>>>> const*, char const*, __va_list_tag*, Thread*, unsigned char*, 
>>>>>>>>>>>>>>>> void*, void*, char const*, int, unsigned long)+0x2c3
>>>>>>>>>>>>>>>> V  [libjvm.so+0x18c56ef]  VMError::report_and_die(Thread*, 
>>>>>>>>>>>>>>>> void*, char const*, int, char const*, char const*, 
>>>>>>>>>>>>>>>> __va_list_tag*)+0x2f
>>>>>>>>>>>>>>>> V  [libjvm.so+0xb55aa0]  report_vm_error(char const*, int, char 
>>>>>>>>>>>>>>>> const*, char const*, ...)+0x100
>>>>>>>>>>>>>>>> V  [libjvm.so+0x11f2cfe]  JvmtiThreadState::cur_stack_depth()+0x14e
>>>>>>>>>>>>>>>> V  [libjvm.so+0x11f3257] 
>>>>>>>>>>>>>>>>  JvmtiThreadState::update_for_pop_top_frame()+0x27
>>>>>>>>>>>>>>>> V  [libjvm.so+0x119af99]  VM_UpdateForPopTopFrame::doit()+0xb9
>>>>>>>>>>>>>>>> V  [libjvm.so+0x1908982]  VM_Operation::evaluate()+0x132
>>>>>>>>>>>>>>>> V  [libjvm.so+0x19040be] 
>>>>>>>>>>>>>>>>  VMThread::evaluate_operation(VM_Operation*) [clone 
>>>>>>>>>>>>>>>> .constprop.51]+0x18e
>>>>>>>>>>>>>>>> V  [libjvm.so+0x1904960]  VMThread::loop()+0x4c0
>>>>>>>>>>>>>>>> V  [libjvm.so+0x1904f53]  VMThread::run()+0xd3
>>>>>>>>>>>>>>>> V  [libjvm.so+0x14e8300]  thread_native_entry(Thread*)+0x100
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> VM_Operation (0x00002af4d8502910): UpdateForPopTopFrame, mode: 
>>>>>>>>>>>>>>>> safepoint, requested by thread 0x00002af4dc008800
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>
>>
> 


More information about the serviceability-dev mailing list