RFR 8249217: Unexpected StackOverflowError in "process reaper" thread still happens

Thu Jul 23 04:00:43 UTC 2020

Hi David,

Yep, that's the focal point.
Its both debug and -Xcomp.
There must be something that changes the timing or the synchronization
or the sizes; but that's more than I know about how the VM works.

Running the TestHumongousNonArrayAllocation test repeatedly does show 
the problem
intermittently but it would take a HS engineer to figure it out.

For now, all I have is an avoidence attempt.

Thanks, Roger

On 7/22/20 9:46 PM, David Holmes wrote:
> On 23/07/2020 11:27 am, Roger Riggs wrote:
>> Hi David,
>>
>> It took some digging in the hs_err files and looking at a core dump 
>> to determine what class was being initialized.
>
> Doh! I knew that [1]:
>
> "So what are we trying to initialize? Unfortunately that's not 
> discernible from the hs_err file, but looking at fullAddCount we would 
> have to suspect this:
>
>   if ((h = ThreadLocalRandom.getProbe()) == 0) { "
>
> but forgot.
>
> Sorry for the noise.
>
> I'm still a bit perplexed with why this issue only seems to arise with 
> debug builds but ...
>
> David
> -----
>
> [1] 
> https://bugs.openjdk.java.net/browse/JDK-8217475?focusedCommentId=14350680&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14350680
>
>>
>> I attached one to the current bug, there are others on the previous bug.
>>
>> The "call_class_initializer" is the hook.
>>
>> Roger
>>
>> Stack: [0x00007f41eeeb8000,0x00007f41eeedc000], 
>> sp=0x00007f41eeed1e80, free space=103k
>> Native frames: (J=compiled Java code, A=aot compiled Java code, 
>> j=interpreted, Vv=VM code, C=native code)
>> V  [libjvm.so+0xa78e24] Exceptions::debug_check_abort_helper(Handle, 
>> char const*)+0x184
>> V  [libjvm.so+0xa79b01]  Exceptions::_throw(Thread*, char const*, 
>> int, Handle, char const*)+0x121
>> V  [libjvm.so+0xa7b69e] 
>> Exceptions::throw_stack_overflow_exception(Thread*, char const*, int, 
>> methodHandle const&)+0xde
>> V  [libjvm.so+0xcfff7f]  JavaCalls::call_helper(JavaValue*, 
>> methodHandle const&, JavaCallArguments*, Thread*)+0x39f
>> V  [libjvm.so+0xcc128f] 
>> InstanceKlass::call_class_initializer(Thread*)+0x1bf
>> V  [libjvm.so+0xcc2787] InstanceKlass::initialize_impl(Thread*)+0x637
>> V  [libjvm.so+0x113093f] LinkResolver::resolve_static_call(CallInfo&, 
>> LinkInfo const&, bool, Thread*)+0xcf
>> V  [libjvm.so+0x11377e3] LinkResolver::resolve_invoke(CallInfo&, 
>> Handle, constantPoolHandle const&, int, Bytecodes::Code, Thread*)+0x183
>> V  [libjvm.so+0x15474e3] 
>> SharedRuntime::find_callee_info_helper(JavaThread*, vframeStream&, 
>> Bytecodes::Code&, CallInfo&, Thread*)+0x5f3
>> V  [libjvm.so+0x15498ae] 
>> SharedRuntime::resolve_sub_helper(JavaThread*, bool, bool, 
>> Thread*)+0x18e
>> V  [libjvm.so+0x1549eae] SharedRuntime::resolve_helper(JavaThread*, 
>> bool, bool, Thread*)+0x4e
>> V  [libjvm.so+0x154a1b1] 
>> SharedRuntime::resolve_static_call_C(JavaThread*)+0x131
>> v  ~RuntimeStub::resolve_static_call
>> J 5841 c1 java.util.concurrent.ConcurrentHashMap.fullAddCount(JZ)V 
>> java.base at 16-internal (462 bytes) @ 0x00007f4209952e4c 
>> [0x00007f4209952de0+0x000000000000006c]
>> J 4202 c2 java.util.concurrent.ConcurrentHashMap.addCount(JI)V 
>> java.base at 16-internal (280 bytes) @ 0x00007f42109cf734 
>> [0x00007f42109cf6e0+0x0000000000000054]
>> J 4213 c1 
>> java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object; 
>> java.base at 16-internal (432 bytes) @ 0x00007f4209a5a374 
>> [0x00007f4209a59800+0x0000000000000b74]
>> J 798 c1 
>> java.lang.invoke.MethodType$ConcurrentWeakInternSet.add(Ljava/lang/Object;)Ljava/lang/Object; 
>> java.base at 16-internal (66 bytes) @ 0x00007f42094af3b4 
>> [0x00007f42094af0a0+0x0000000000000314]
>> J 880 c1 
>> java.lang.invoke.MethodType.makeImpl(Ljava/lang/Class;[Ljava/lang/Class;Z)Ljava/lang/invoke/MethodType; 
>> java.base at 16-internal (108 bytes) @ 0x00007f42094e2bcc 
>> [0x00007f42094e2720+0x00000000000004ac]
>> J 1001 c1 
>> java.lang.invoke.MethodHandleNatives.findMethodHandleType(Ljava/lang/Class;[Ljava/lang/Class;)Ljava/lang/invoke/MethodType; 
>> java.base at 16-internal (7 bytes) @ 0x00007f420955997c 
>> [0x00007f4209559940+0x000000000000003c]
>> v  ~StubRoutines::call_stub
>> V  [libjvm.so+0xd0020a]  JavaCalls::call_helper(JavaValue*, 
>> methodHandle const&, JavaCallArguments*, Thread*)+0x62a
>> V  [libjvm.so+0xd017e2]  JavaCalls::call_static(JavaValue*, Klass*, 
>> Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x312
>> V  [libjvm.so+0x166c2e1] 
>> SystemDictionary::find_method_handle_type(Symbol*, Klass*, 
>> Thread*)+0x1541
>> V  [libjvm.so+0x166e31f] 
>> SystemDictionary::find_method_handle_invoker(Klass*, Symbol*, 
>> Symbol*, Klass*, Handle*, Thread*)+0x5f
>> V  [libjvm.so+0x112a15b] 
>> LinkResolver::lookup_polymorphic_method(LinkInfo const&, Handle*, 
>> Thread*)+0x52b
>> V  [libjvm.so+0x112a6a7] LinkResolver::resolve_handle_call(CallInfo&, 
>> LinkInfo const&, Thread*)+0xa7
>> V  [libjvm.so+0x112a944] 
>> LinkResolver::resolve_invokehandle(CallInfo&, constantPoolHandle 
>> const&, int, Thread*)+0x184
>> V  [libjvm.so+0x11376c9] LinkResolver::resolve_invoke(CallInfo&, 
>> Handle, constantPoolHandle const&, int, Bytecodes::Code, Thread*)+0x69
>> V  [libjvm.so+0x727eec]  Runtime1::patch_code(JavaThread*, 
>> Runtime1::StubID)+0x156c
>> V  [libjvm.so+0x72a847] 
>> Runtime1::move_appendix_patching(JavaThread*)+0x37
>> v  ~RuntimeStub::load_appendix_patching Runtime1 stub
>> J 5283 c1 java.lang.ProcessHandleImpl$1.run()V java.base at 16-internal 
>> (136 bytes) @ 0x00007f420946c262 [0x00007f420946b3c0+0x0000000000000ea2]
>> J 5275 c1 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V 
>> java.base at 16-internal (187 bytes) @ 0x00007f420946d91c 
>> [0x00007f420946d680+0x000000000000029c]
>> J 5272 c1 java.util.concurrent.ThreadPoolExecutor$Worker.run()V 
>> java.base at 16-internal (9 bytes) @ 0x00007f4209916c7c 
>> [0x00007f4209916be0+0x000000000000009c]
>> J 3308 c1 java.lang.Thread.run()V java.base at 16-internal (17 bytes) @ 
>> 0x00007f42097c2bf4 [0x00007f42097c2b60+0x0000000000000094]
>> v  ~StubRoutines::call_stub
>> V  [libjvm.so+0xd0020a]  JavaCalls::call_helper(JavaValue*, 
>> methodHandle const&, JavaCallArguments*, Thread*)+0x62a
>> V  [libjvm.so+0xd00b15]  JavaCalls::call_virtual(JavaValue*, Klass*, 
>> Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x4b5
>> V  [libjvm.so+0xd00fc1]  JavaCalls::call_virtual(JavaValue*, Handle, 
>> Klass*, Symbol*, Symbol*, Thread*)+0xb1
>> V  [libjvm.so+0xe7ed63]  thread_entry(JavaThread*, Thread*)+0x123
>> V  [libjvm.so+0x16d3f9c]  JavaThread::thread_main_inner()+0x21c
>> V  [libjvm.so+0x16d9d20]  Thread::call_run()+0x100
>> V  [libjvm.so+0x13ddbe6]  thread_native_entry(Thread*)+0x116
>>
>>
>>
>>
>> On 7/22/20 9:16 PM, David Holmes wrote:
>>> Hi Roger,
>>>
>>> On 23/07/2020 12:51 am, Roger Riggs wrote:
>>>> Please review a change to the Process reaper thread initialization 
>>>> to preemptively
>>>> make sure classes ThreadLocalRandom and ConcurrentHashMap are 
>>>> initialized.
>>>
>>> I don't see ThreadLocalRandom  appearing in any of the stack traces. ??
>>>
>>> David
>>> -----
>>>
>>>>  From the stack overflow failures, it seems that the classes have 
>>>> not been initialized
>>>> before they are used during processing the termination of a process.
>>>> When the initialization is performed on the smaller reaper stack, 
>>>> it occasionally
>>>> exceeds the available stack.
>>>>
>>>> As an aid to diagnostics, 
>>>> -XX:AbortVMOnException=java.lang.StackOverflowError
>>>> is added to TestHumongousNonArrayAllocation that has failed 
>>>> intermittently.
>>>> If the problem happens again it will produce an hs_error file with 
>>>> useful details
>>>> and will otherwise not change the test behavior.
>>>>
>>>> Webrev:
>>>> http://cr.openjdk.java.net/~rriggs/webrev-stackoverflow-8249217/
>>>>
>>>> Issue:
>>>>    https://bugs.openjdk.java.net/browse/JDK-8249217
>>>>
>>>> Thanks, Roger
>>>>
>>