The failure
serguei.spitsyn at oracle.com
serguei.spitsyn at oracle.com
Tue Oct 23 23:15:43 UTC 2018
Hi David and Robbin,
On 10/23/18 7:38 AM, Robbin Ehn wrote:
> Hi,
>
> On 10/23/18 10:34 AM, David Holmes wrote:
>> Hi Serguei,
>>
>> The VMThread is executing VM_HandshakeAllThreads which is not a
>> safepoint operation. There's no real way to tell from the stacks what
>> it's stuck on.
Good point.
We agreed with Leonid that he will try to reproduce this deadlock with
fewer kitchensink modules and modes.
It would really help if there are less threads to analyze.
> I cannot find a thread that is not considered safepoint safe or
> is_ext_suspended (thread 146). So the handshake should go through. The
> handshake will log a warning after a while, is there such warning from
> the handshake operation?
There is Thread #136 that executes JvmtiSuspendControl::suspend().
Not sure if it helps.
>
> There are several threads competing with e.g. Threads_lock, and
> threads waiting for GC and several other VM ops, could it just be
> really slow?
No idea yet.
Thanks,
Serguei
> /Robbin
>
>>
>> David
>>
>> On 23/10/2018 5:58 PM, serguei.spitsyn at oracle.com wrote:
>>> Hi David,
>>>
>>> You are right, thanks.
>>> It means, this deadlock needs more analysis.
>>> For completeness, the stack traces are in attachments.
>>>
>>> Thanks,
>>> Serguei
>>>
>>>
>>> On 10/23/18 00:43, David Holmes wrote:
>>>> Hi Serguei,
>>>>
>>>> The JvmtiThreadState_lock is always acquired with safepoint checks
>>>> enabled, so all JavaThreads blocked trying to acquire it will be
>>>> _thread_blocked and so safepoint-safe and so won't be holding up
>>>> the safepoint.
>>>>
>>>> David
>>>>
>>>> On 23/10/2018 5:21 PM, serguei.spitsyn at oracle.com wrote:
>>>>> Hi,
>>>>>
>>>>> I've added the seviceability-dev mailing list.
>>>>> It can be interesting for the SVC folks. :)
>>>>>
>>>>>
>>>>> On 10/22/18 22:14, Leonid Mesnik wrote:
>>>>>> Hi
>>>>>>
>>>>>> Seems last version also crashes with 2 other different symptoms.
>>>>>> http://java.se.oracle.com:10065/mdash/jobs/lmesnik-ks8-20181021-0638-7157/results?search=status%3Afailed+AND+-state%3Ainvalid
>>>>>> <http://java.se.oracle.com:10065/mdash/jobs/lmesnik-ks8-20181021-0638-7157/results?search=status:failed+AND+-state:invalid>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Also it might hangs with stack attached. Seems that test might
>>>>>> be blocked because it invoke 2 jvmti methods. Can jvmti agent
>>>>>> invoke jvmti methods from different threads?
>>>>>
>>>>> Yes, in general.
>>>>> However, you have to be careful when using debugging features.
>>>>> Below, one thread is enabling single stepping while another thread
>>>>> is being suspended.
>>>>> Both are blocked at a safepoint which is Okay in general but not
>>>>> Okay if they hold any lock.
>>>>> For instance, the thread #152 is holding the monitor
>>>>> JvmtiThreadState.
>>>>>
>>>>> Also, I see a couple of more threads that are interesting as well:
>>>>>
>>>>> Thread 159 (Thread 0x2ae40b78f700 (LWP 27962)):
>>>>> #0 0x00002ae3927b5945 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>>> /lib64/libpthread.so.0
>>>>> #1 0x00002ae393ba8d63 in os::PlatformEvent::park
>>>>> (this=this at entry=0x2ae3984c9100) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897
>>>>> #2 0x00002ae393b50920 in ParkCommon (timo=0, ev=0x2ae3984c9100)
>>>>> at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399
>>>>> #3 Monitor::ILock (this=0x2ae398024f10, Self=0x2ae3984c7800) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:461
>>>>> #4 0x00002ae393b512c1 in lock
>>>>> (Self=0x2ae3984c7is_ext_suspended800, this=0x2ae398024f10) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:910
>>>>> #5 Monitor::lock (this=this at entry=0x2ae398024f10) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:919
>>>>> #6 0x00002ae39350510c in MutexLocker (mutex=0x2ae398024f10,
>>>>> this=<synthetic pointer>) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutexLocker.hpp:182
>>>>>
>>>>> #7 ciEnv::cache_jvmti_state (this=this at entry=0x2ae40b78eb30) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/ci/ciEnv.cpp:229
>>>>>
>>>>> #8 0x00002ae3935d3294 in CompileBroker::invoke_compiler_on_method
>>>>> (task=task at entry=0x2ae48800ff40) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/compiler/compileBroker.cpp:2084
>>>>>
>>>>> #9 0x00002ae3935d4f48 in CompileBroker::compiler_thread_loop ()
>>>>> at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/compiler/compileBroker.cpp:1798
>>>>>
>>>>> #10 0x00002ae393d7338a in JavaThread::thread_main_inner
>>>>> (this=this at entry=0x2ae3984c7800) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1795
>>>>> #11 0x00002ae393d736c6 in JavaThread::run (this=0x2ae3984c7800) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1775
>>>>> #12 0x00002ae393ba0070 in thread_native_entry
>>>>> (thread=0x2ae3984c7800) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/linux/os_linux.cpp:698
>>>>> #13 0x00002ae3927b1e25 in start_thread () from /lib64/libpthread.so.0
>>>>> #14 0x00002ae392cc234d in clone () from /lib64/libc.so.6
>>>>>
>>>>> Thread 158 (Thread 0x2ae40b890700 (LWP 27963)):
>>>>> #0 0x00002ae3927b5945 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>>> /lib64/libpthread.so.0
>>>>> #1 0x00002ae393ba8d63 in os::PlatformEvent::park
>>>>> (this=this at entry=0x2ae3984cbb00) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897
>>>>> #2 0x00002ae393b50920 in ParkCommon (timo=0, ev=0x2ae3984cbb00)
>>>>> at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399
>>>>> #3 Monitor::ILock (this=0x2ae398024f10, Self=0x2ae3984ca800) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:461
>>>>> #4 0x00002ae393b512c1 in lock (Self=0x2ae3984ca800,
>>>>> this=0x2ae398024f10) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:910
>>>>> #5 Monitor::lock (this=this at entry=0x2ae398024f10) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:919
>>>>> #6 0x00002ae39350510c in MutexLocker (mutex=0x2ae398024f10,
>>>>> this=<synthetic pointer>) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutexLocker.hpp:182
>>>>>
>>>>> #7 ciEnv::cache_jvmti_state (this=this at entry=0x2ae40b88fb30) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/ci/ciEnv.cpp:229
>>>>>
>>>>> #8 0x00002ae3935d3294 in CompileBroker::invoke_compiler_on_method
>>>>> (task=task at entry=0x2ae49c00a670) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/compiler/compileBroker.cpp:2084
>>>>>
>>>>> #9 0x00002ae3935d4f48 in CompileBroker::compiler_thread_loop ()
>>>>> at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/compiler/compileBroker.cpp:1798
>>>>>
>>>>> #10 0x00002ae393d7338a in JavaThread::thread_main_inner
>>>>> (this=this at entry=0x2ae3984ca800) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1795
>>>>> #11 0x00002ae393d736c6 in JavaThread::run (this=0x2ae3984ca800) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1775
>>>>> #12 0x00002ae393ba0070 in thread_native_entry
>>>>> (thread=0x2ae3984ca800) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/linux/os_linux.cpp:698
>>>>> #13 0x00002ae3927b1e25 in start_thread () from /lib64/libpthread.so.0
>>>>> #14 0x00002ae392cc234d in clone () from /lib64/libc.so.6
>>>>>
>>>>> Thread 51 (Thread 0x2ae49549b700 (LWP 29678)):
>>>>> #0 0x00002ae3927b5945 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>>> /lib64/libpthread.so.0
>>>>> #1 0x00002ae393ba8d63 in os::PlatformEvent::park
>>>>> (this=this at entry=0x2ae460061c00) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897
>>>>> #2 0x00002ae393b50920 in ParkCommon (timo=0, ev=0x2ae460061c00)
>>>>> at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399
>>>>> #3 Monitor::ILock (this=0x2ae398024f10, Self=0x2ae4600c2800) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:461
>>>>> #4 0x00002ae393b512c1 in lock (Self=0x2ae4600c2800,
>>>>> this=0x2ae398024f10) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:910
>>>>> #5 Monitor::lock (this=this at entry=0x2ae398024f10) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:919
>>>>> #6 0x00002ae393999682 in MutexLocker (mutex=0x2ae398024f10,
>>>>> this=<synthetic pointer>) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutexLocker.hpp:182
>>>>>
>>>>> #7 thread_started (thread=0x2ae4600c2800) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEventController.cpp:668
>>>>>
>>>>> #8 JvmtiEventController::thread_started (thread=0x2ae4600c2800)
>>>>> at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEventController.cpp:1027
>>>>>
>>>>> #9 0x00002ae39399f3a0 in JvmtiExport::post_thread_start
>>>>> (thread=thread at entry=0x2ae4600c2800) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiExport.cpp:1395
>>>>>
>>>>> #10 0x00002ae393d737d8 in JavaThread::run (this=0x2ae4600c2800) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1764
>>>>> #11 0x00002ae393ba0070 in thread_native_entry
>>>>> (thread=0x2ae4600c2800) at
>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/linux/os_linux.cpp:698
>>>>> #12 0x00002ae3927b1e25 in start_thread () from /lib64/libpthread.so.0
>>>>> #13 0x00002ae392cc234d in clone () from /lib64/libc.so.6
>>>>>
>>>>>
>>>>> These two thread are blocked on the monitor JvmtiThreadState_lock
>>>>> in the function ciEnv::cache_jvmti_state().
>>>>> Also, there are many threads (like #51) that are executing
>>>>> JvmtiExport::post_thread_start and blocked on the same monitor.
>>>>>
>>>>>
>>>>> Now, the question is why this safepoint can not start?
>>>>> What thread is blocking it? Or in reverse, what thread this
>>>>> safepoint is waiting for?
>>>>>
>>>>> I think, this safepoint operation is waiting for all threads that
>>>>> are blocked on the JvmtiThreadState_lock.
>>>>>
>>>>> Conclusion:
>>>>>
>>>>> The deadlock is:
>>>>>
>>>>> Thread #152:
>>>>> - grabbed the monitor JvmtiThreadState_lock
>>>>> - blocked in the VM_GetCurrentLocation in the function
>>>>> JvmtiEnvThreadState::reset_current_location()
>>>>>
>>>>> Many other threads:
>>>>> - blocked on the monitor JvmtiThreadState_lock
>>>>> - can not reach the blocked at a safepoint state (all threads
>>>>> have to reach this state for this safepoint to happen)
>>>>>
>>>>> It seems to me, this is a bug which has to be filed.
>>>>>
>>>>> My guess is that this will stop to reproduce after if you turn off
>>>>> the single stepping for thread #152.
>>>>> Please, let me know about the results.
>>>>>
>>>>>
>>>>>> Assuming that crashes look like VM bugs I think it make sense to
>>>>>> integrate jvmti changes but *don't* enabled jvmti module by default.
>>>>>
>>>>> This one is a deadlock.
>>>>> However, the root cause is a race condition that can potentially
>>>>> result in both deadlocks and crashes.
>>>>> So, I'm curious if you observed crashes as well.
>>>>>
>>>>>
>>>>>> And add to more tests with jvmti enabled.
>>>>>> So anyone could easily run them to reproduce crashes. This test
>>>>>> would be out of CI to don't introduce any bugs. Does it make sense?
>>>>>>
>>>>>> Consider hang - I think that it might be product bug since I
>>>>>> don't see any locking on my monitors. But I am not sure. Is it
>>>>>> possible that any my code jvmti agent prevent VM to get into
>>>>>> safepoint?
>>>>>> Could we discuss it tomorrow or his week when you have a time?
>>>>>
>>>>> Yes, of course.
>>>>> Let's find some time tomorrow.
>>>>>
>>>>>
>>>>>> Any suggestion how to diagnose deadlock would be great.
>>>>>
>>>>> Analysis of stack traces is needed.
>>>>> It is non-trivial in this particular case as there are so many
>>>>> threads executed at the same time.
>>>>>
>>>>>
>>>>>> Part of stack trace with 2 my threads only:
>>>>>>
>>>>>> Thread 136 (Thread 0x2ae494100700 (LWP 28023)):
>>>>>> #0 0x00002ae3927b5945 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>>>> /lib64/libpthread.so.0
>>>>>> #1 0x00002ae393ba8d63 in os::PlatformEvent::park
>>>>>> (this=this at entry=0x2ae454005800) at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897
>>>>>> #2 0x00002ae393b50cf8 in ParkCommon (timo=0, ev=0x2ae454005800)
>>>>>> at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399
>>>>>> #3 Monitor::IWait (this=this at entry=0x2ae398023c10,
>>>>>> Self=Self at entry=0x2ae454004800, timo=timo at entry=0) at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:76\
>>>>>> 8
>>>>>> #4 0x00002ae393b51f2e in Monitor::wait
>>>>>> (this=this at entry=0x2ae398023c10, no_safepoint_check=<optimized
>>>>>> out>, timeout=timeout at entry=0,
>>>>>> as_suspend_equivalent=as_suspend_equivalent at en\
>>>>>> try=false) at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:1106
>>>>>> #5 0x00002ae393de7867 in VMThread::execute
>>>>>> (op=op at entry=0x2ae4940ffb10) at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/vmThread.cpp:657
>>>>>> #6 0x00002ae393d6a3bd in JavaThread::java_suspend
>>>>>> (this=this at entry=0x2ae3985f2000) at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:2321
>>>>>> #7 0x00002ae3939ad7e1 in JvmtiSuspendControl::suspend
>>>>>> (java_thread=java_thread at entry=0x2ae3985f2000) at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiImpl.cpp:8\
>>>>>> 47
>>>>>> #8 0x00002ae3939887ae in JvmtiEnv::SuspendThread
>>>>>> (this=this at entry=0x2ae39801b270, java_thread=0x2ae3985f2000) at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiE\
>>>>>> nv.cpp:955
>>>>>> #9 0x00002ae39393a8c6 in jvmti_SuspendThread
>>>>>> (env=0x2ae39801b270, thread=0x2ae49929fdf8) at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/build/linux-x64/hotspot/variant-server/gensrc/jvmtifiles\
>>>>>>
>>>>>> /jvmtiEnter.cpp:527
>>>>>> #10 0x00002ae394d973ee in agent_sampler (jvmti=0x2ae39801b270,
>>>>>> env=<optimized out>, p=<optimized out>) at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/closed/test/hotspot/jtreg/applications/kitc\
>>>>>> hensink/process/stress/modules/libJvmtiStressModule.c:274
>>>>>> #11 0x00002ae3939ab24d in call_start_function
>>>>>> (this=0x2ae454004800) at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiImpl.cpp:85
>>>>>> #12 JvmtiAgentThread::start_function_wrapper
>>>>>> (thread=0x2ae454004800, __the_thread__=<optimized out>) at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiImpl.cpp:79
>>>>>> #13 0x00002ae393d7338a in JavaThread::thread_main_inner
>>>>>> (this=this at entry=0x2ae454004800) at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1795
>>>>>> #14 0x00002ae393d736c6 in JavaThread::run (this=0x2ae454004800)
>>>>>> at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/thread.cpp:1775
>>>>>> #15 0x00002ae393ba0070 in thread_native_entry
>>>>>> (thread=0x2ae454004800) at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/linux/os_linux.cpp:698
>>>>>> #16 0x00002ae3927b1e25 in start_thread () from
>>>>>> /lib64/libpthread.so.0
>>>>>> #17 0x00002ae392cc234d in clone () from /lib64/libc.so.6
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thread 152 (Thread 0x2ae427060700 (LWP 27995)):
>>>>>> #0 0x00002ae3927b5945 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>>>>> /lib64/libpthread.so.0
>>>>>> #1 0x00002ae393ba8d63 in os::PlatformEvent::park
>>>>>> (this=this at entry=0x2ae3985e7400) at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/os/posix/os_posix.cpp:1897
>>>>>> #2 0x00002ae393b50cf8 in ParkCommon (timo=0, ev=0x2ae3985e7400)
>>>>>> at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:399
>>>>>> #3 Monitor::IWait (this=this at entry=0x2ae398023c10,
>>>>>> Self=Self at entry=0x2ae3985e6000, timo=timo at entry=0) at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:76\
>>>>>> 8
>>>>>> #4 0x00002ae393b51f2e in Monitor::wait
>>>>>> (this=this at entry=0x2ae398023c10, no_safepoint_check=<optimized
>>>>>> out>, timeout=timeout at entry=0,
>>>>>> as_suspend_equivalent=as_suspend_equivalent at en\
>>>>>> try=false) at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/mutex.cpp:1106
>>>>>> #5 0x00002ae393de7867 in VMThread::execute (op=0x2ae42705f500)
>>>>>> at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/runtime/vmThread.cpp:657
>>>>>> #6 0x00002ae3939965f3 in
>>>>>> JvmtiEnvThreadState::reset_current_location
>>>>>> (this=this at entry=0x2ae6bc000d80,
>>>>>> event_type=event_type at entry=JVMTI_EVENT_SINGLE_STEP,
>>>>>> enabled=enabled at entry=tr\
>>>>>> ue) at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEnvThreadState.cpp:312
>>>>>>
>>>>>> #7 0x00002ae393997acf in recompute_env_thread_enabled
>>>>>> (state=0x2ae6bc000cd0, ets=0x2ae6bc000d80) at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEventControlle\
>>>>>>
>>>>>> r.cpp:490
>>>>>> #8 JvmtiEventControllerPrivate::recompute_thread_enabled
>>>>>> (state=state at entry=0x2ae6bc000cd0) at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEventController.cpp\
>>>>>>
>>>>>> :523
>>>>>> #9 0x00002ae393998168 in
>>>>>> JvmtiEventControllerPrivate::recompute_enabled () at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEventController.cpp:598
>>>>>>
>>>>>> #10 0x00002ae39399a244 in set_user_enabled (enabled=true,
>>>>>> event_type=JVMTI_EVENT_SINGLE_STEP, thread=0x0,
>>>>>> env=0x2ae39801b270) at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/sha\
>>>>>> re/prims/jvmtiEventController.cpp:818
>>>>>> #11 JvmtiEventController::set_user_enabled (env=0x2ae39801b270,
>>>>>> thread=0x0, event_type=JVMTI_EVENT_SINGLE_STEP,
>>>>>> enabled=<optimized out>) at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/\
>>>>>> hotspot/share/prims/jvmtiEventController.cpp:963
>>>>>> #12 0x00002ae393987d2d in JvmtiEnv::SetEventNotificationMode
>>>>>> (this=this at entry=0x2ae39801b270, mode=mode at entry=JVMTI_ENABLE,
>>>>>> event_type=event_type at entry=JVMTI_EVENT_SINGLE_STEP, eve\
>>>>>> nt_thread=event_thread at entry=0x0) at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiEnv.cpp:543
>>>>>> #13 0x00002ae3939414eb in jvmti_SetEventNotificationMode
>>>>>> (env=0x2ae39801b270, mode=mode at entry=JVMTI_ENABLE,
>>>>>> event_type=event_type at entry=JVMTI_EVENT_SINGLE_STEP,
>>>>>> event_thread=event_\
>>>>>> thread at entry=0x0) at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/build/linux-x64/hotspot/variant-server/gensrc/jvmtifiles/jvmtiEnter.cpp:5389
>>>>>>
>>>>>> #14 0x00002ae394d97989 in enable_events () at
>>>>>> /scratch/lmesnik/ws/hs-bigapps/closed/test/hotspot/jtreg/applications/kitchensink/process/stress/modules/libJvmtiStressModule.c:519
>>>>>>
>>>>>> #15 0x00002ae394d98070 in
>>>>>> Java_applications_kitchensink_process_stress_modules_JvmtiStressModule_startIteration
>>>>>> (env=<optimized out>, this=<optimized out>) at
>>>>>> /scratch/lmesnik/ws/h\
>>>>>> s-bigapps/closed/test/hotspot/jtreg/applications/kitchensink/process/stress/modules/libJvmtiStressModule.c:697
>>>>>>
>>>>>> #16 0x00002ae3a43ef257 in ?? ()
>>>>>> #17 0x00002ae3a43eede1 in ?? ()
>>>>>> #18 0x00002ae42705f878 in ?? ()
>>>>>> #19 0x00002ae40ad334e0 in ?? ()
>>>>>> #20 0x00002ae42705f8e0 in ?? ()
>>>>>> #21 0x00002ae40ad33c68 in ?? ()
>>>>>> #22 0x0000000000000000 in ?? ()
>>>>>
>>>>> Thanks,
>>>>> Serguei
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Leonid
>>>>>>
>>>>>>
>>>>>>> On Oct 9, 2018, at 4:52 PM, serguei.spitsyn at oracle.com
>>>>>>> <mailto:serguei.spitsyn at oracle.com> wrote:
>>>>>>>
>>>>>>> Hi Leonid,
>>>>>>>
>>>>>>> There is an existing bug:
>>>>>>> https://bugs.openjdk.java.net/browse/JDK-8043571
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Serguei
>>>>>>>
>>>>>>>
>>>>>>> On 10/9/18 16:11, Leonid Mesnik wrote:
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> During fixing kitchensink I get
>>>>>>>> assert(_cur_stack_depth == count_frames()) failed:
>>>>>>>> cur_stack_depth out of sync
>>>>>>>>
>>>>>>>> Do you know if i might be bug in my jvmti agent?
>>>>>>>>
>>>>>>>> Leonid
>>>>>>>>
>>>>>>>>
>>>>>>>> #
>>>>>>>> # A fatal error has been detected by the Java Runtime Environment:
>>>>>>>> #
>>>>>>>> # Internal Error
>>>>>>>> (/scratch/lmesnik/ws/hs-bigapps/open/src/hotspot/share/prims/jvmtiThreadState.cpp:277),
>>>>>>>> pid=13926, tid=13962
>>>>>>>> # assert(_cur_stack_depth == count_frames()) failed:
>>>>>>>> cur_stack_depth out of sync
>>>>>>>> #
>>>>>>>> # JRE version: Java(TM) SE Runtime Environment (12.0)
>>>>>>>> (fastdebug build
>>>>>>>> 12-internal+0-2018-10-08-2342517.lmesnik.hs-bigapps)
>>>>>>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug
>>>>>>>> 12-internal+0-2018-10-08-2342517.lmesnik.hs-bigapps, mixed
>>>>>>>> mode, tiered, compressed oops, g1 gc, linux-amd64)
>>>>>>>> # Core dump will be written. Default location: Core dumps may
>>>>>>>> be processed with "/usr/libexec/abrt-hook-ccpp %s %c %p %u %g
>>>>>>>> %t e %P %I %h" (or dumping to
>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_KitchensinkSanity_java/scratch/0/core.13926)
>>>>>>>>
>>>>>>>> #
>>>>>>>> # If you would like to submit a bug report, please visit:
>>>>>>>> # http://bugreport.java.com/bugreport/crash.jsp
>>>>>>>> #
>>>>>>>>
>>>>>>>> --------------- S U M M A R Y ------------
>>>>>>>>
>>>>>>>> Command Line: -XX:MaxRAMPercentage=2 -XX:MaxRAMPercentage=50
>>>>>>>> -XX:+CrashOnOutOfMemoryError
>>>>>>>> -Djava.net.preferIPv6Addresses=false -XX:-PrintVMOptions
>>>>>>>> -XX:+DisplayVMOutputToStderr -XX:+UsePerfData
>>>>>>>> -Xlog:gc*,gc+heap=debug:gc.log:uptime,timemillis,level,tags
>>>>>>>> -XX:+DisableExplicitGC -XX:+PrintFlagsFinal
>>>>>>>> -XX:+StartAttachListener -XX:NativeMemoryTracking=detail
>>>>>>>> -XX:+FlightRecorder
>>>>>>>> --add-exports=java.base/java.lang=ALL-UNNAMED
>>>>>>>> --add-opens=java.base/java.lang=ALL-UNNAMED
>>>>>>>> --add-exports=java.xml/com.sun.org.apache.xerces.internal.parsers=ALL-UNNAMED
>>>>>>>> --add-exports=java.xml/com.sun.org.apache.xerces.internal.util=ALL-UNNAMED
>>>>>>>> -Djava.io.tmpdir=/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_KitchensinkSanity_java/scratch/0/java.io.tmpdir
>>>>>>>> -Duser.home=/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_KitchensinkSanity_java/scratch/0/user.home
>>>>>>>> -agentpath:/scratch/lmesnik/ws/hs-bigapps/build/linux-x64/images/test/hotspot/jtreg/native/libJvmtiStressModule.so
>>>>>>>> applications.kitchensink.process.stress.Main
>>>>>>>> /scratch/lmesnik/ws/hs-bigapps/build/linux-x64/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_KitchensinkSanity_java/scratch/0/kitchensink.final.properties
>>>>>>>>
>>>>>>>>
>>>>>>>> Host: scaaa118.us.oracle.com <http://scaaa118.us.oracle.com>,
>>>>>>>> Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz, 32 cores, 235G,
>>>>>>>> Oracle Linux Server release 7.3
>>>>>>>> Time: Tue Oct 9 16:06:07 2018 PDT elapsed time: 31 seconds (0d
>>>>>>>> 0h 0m 31s)
>>>>>>>>
>>>>>>>> --------------- T H R E A D ---------------
>>>>>>>>
>>>>>>>> Current thread (0x00002af3dc6ac800): VMThread "VM Thread"
>>>>>>>> [stack: 0x00002af44f10a000,0x00002af44f20a000] [id=13962]
>>>>>>>> _threads_hazard_ptr=0x00002af4ac090eb0,
>>>>>>>> _nested_threads_hazard_ptr_cnt=0
>>>>>>>>
>>>>>>>> Stack: [0x00002af44f10a000,0x00002af44f20a000],
>>>>>>>> sp=0x00002af44f208720, free space=1017k
>>>>>>>> Native frames: (J=compiled Java code, A=aot compiled Java code,
>>>>>>>> j=interpreted, Vv=VM code, C=native code)
>>>>>>>> V [libjvm.so+0x18c4923] VMError::report_and_die(int, char
>>>>>>>> const*, char const*, __va_list_tag*, Thread*, unsigned char*,
>>>>>>>> void*, void*, char const*, int, unsigned long)+0x2c3
>>>>>>>> V [libjvm.so+0x18c56ef] VMError::report_and_die(Thread*,
>>>>>>>> void*, char const*, int, char const*, char const*,
>>>>>>>> __va_list_tag*)+0x2f
>>>>>>>> V [libjvm.so+0xb55aa0] report_vm_error(char const*, int, char
>>>>>>>> const*, char const*, ...)+0x100
>>>>>>>> V [libjvm.so+0x11f2cfe]
>>>>>>>> JvmtiThreadState::cur_stack_depth()+0x14e
>>>>>>>> V [libjvm.so+0x11f3257]
>>>>>>>> JvmtiThreadState::update_for_pop_top_frame()+0x27
>>>>>>>> V [libjvm.so+0x119af99] VM_UpdateForPopTopFrame::doit()+0xb9
>>>>>>>> V [libjvm.so+0x1908982] VM_Operation::evaluate()+0x132
>>>>>>>> V [libjvm.so+0x19040be]
>>>>>>>> VMThread::evaluate_operation(VM_Operation*) [clone
>>>>>>>> .constprop.51]+0x18e
>>>>>>>> V [libjvm.so+0x1904960] VMThread::loop()+0x4c0
>>>>>>>> V [libjvm.so+0x1904f53] VMThread::run()+0xd3
>>>>>>>> V [libjvm.so+0x14e8300] thread_native_entry(Thread*)+0x100
>>>>>>>>
>>>>>>>> VM_Operation (0x00002af4d8502910): UpdateForPopTopFrame, mode:
>>>>>>>> safepoint, requested by thread 0x00002af4dc008800
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
More information about the serviceability-dev
mailing list