RFR 8249217: Unexpected StackOverflowError in "process reaper" thread still happens

Thu Jul 23 04:24:26 UTC 2020

Hi,

On Thu, Jul 23, 2020 at 6:01 AM Roger Riggs <Roger.Riggs at oracle.com> wrote:

> Hi David,
>
> Yep, that's the focal point.
> Its both debug and -Xcomp.
> There must be something that changes the timing or the synchronization
> or the sizes; but that's more than I know about how the VM works.
>

Just a really wild guess but:
- debug code may have more C++ frames on stack since we have no inlining
- compiled code may do scalar replacement so we may use more stack slots.

I have similar thoughts about IBMs upcoming c2 scalar replacement
prototype; while the work is cool, it may shake a few tests loose which
rely on exact thread stack sizes.

Cheers, Thomas

>
> Running the TestHumongousNonArrayAllocation test repeatedly does show
> the problem
> intermittently but it would take a HS engineer to figure it out.
>
> For now, all I have is an avoidence attempt.
>
> Thanks, Roger
>
>
> On 7/22/20 9:46 PM, David Holmes wrote:
> > On 23/07/2020 11:27 am, Roger Riggs wrote:
> >> Hi David,
> >>
> >> It took some digging in the hs_err files and looking at a core dump
> >> to determine what class was being initialized.
> >
> > Doh! I knew that [1]:
> >
> > "So what are we trying to initialize? Unfortunately that's not
> > discernible from the hs_err file, but looking at fullAddCount we would
> > have to suspect this:
> >
> >   if ((h = ThreadLocalRandom.getProbe()) == 0) { "
> >
> > but forgot.
> >
> > Sorry for the noise.
> >
> > I'm still a bit perplexed with why this issue only seems to arise with
> > debug builds but ...
> >
> > David
> > -----
> >
> > [1]
> >
> https://bugs.openjdk.java.net/browse/JDK-8217475?focusedCommentId=14350680&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14350680
> >
> >>
> >> I attached one to the current bug, there are others on the previous bug.
> >>
> >> The "call_class_initializer" is the hook.
> >>
> >> Roger
> >>
> >> Stack: [0x00007f41eeeb8000,0x00007f41eeedc000],
> >> sp=0x00007f41eeed1e80, free space=103k
> >> Native frames: (J=compiled Java code, A=aot compiled Java code,
> >> j=interpreted, Vv=VM code, C=native code)
> >> V  [libjvm.so+0xa78e24] Exceptions::debug_check_abort_helper(Handle,
> >> char const*)+0x184
> >> V  [libjvm.so+0xa79b01]  Exceptions::_throw(Thread*, char const*,
> >> int, Handle, char const*)+0x121
> >> V  [libjvm.so+0xa7b69e]
> >> Exceptions::throw_stack_overflow_exception(Thread*, char const*, int,
> >> methodHandle const&)+0xde
> >> V  [libjvm.so+0xcfff7f]  JavaCalls::call_helper(JavaValue*,
> >> methodHandle const&, JavaCallArguments*, Thread*)+0x39f
> >> V  [libjvm.so+0xcc128f]
> >> InstanceKlass::call_class_initializer(Thread*)+0x1bf
> >> V  [libjvm.so+0xcc2787] InstanceKlass::initialize_impl(Thread*)+0x637
> >> V  [libjvm.so+0x113093f] LinkResolver::resolve_static_call(CallInfo&,
> >> LinkInfo const&, bool, Thread*)+0xcf
> >> V  [libjvm.so+0x11377e3] LinkResolver::resolve_invoke(CallInfo&,
> >> Handle, constantPoolHandle const&, int, Bytecodes::Code, Thread*)+0x183
> >> V  [libjvm.so+0x15474e3]
> >> SharedRuntime::find_callee_info_helper(JavaThread*, vframeStream&,
> >> Bytecodes::Code&, CallInfo&, Thread*)+0x5f3
> >> V  [libjvm.so+0x15498ae]
> >> SharedRuntime::resolve_sub_helper(JavaThread*, bool, bool,
> >> Thread*)+0x18e
> >> V  [libjvm.so+0x1549eae] SharedRuntime::resolve_helper(JavaThread*,
> >> bool, bool, Thread*)+0x4e
> >> V  [libjvm.so+0x154a1b1]
> >> SharedRuntime::resolve_static_call_C(JavaThread*)+0x131
> >> v  ~RuntimeStub::resolve_static_call
> >> J 5841 c1 java.util.concurrent.ConcurrentHashMap.fullAddCount(JZ)V
> >> java.base at 16-internal (462 bytes) @ 0x00007f4209952e4c
> >> [0x00007f4209952de0+0x000000000000006c]
> >> J 4202 c2 java.util.concurrent.ConcurrentHashMap.addCount(JI)V
> >> java.base at 16-internal (280 bytes) @ 0x00007f42109cf734
> >> [0x00007f42109cf6e0+0x0000000000000054]
> >> J 4213 c1
> >>
> java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object;
>
> >> java.base at 16-internal (432 bytes) @ 0x00007f4209a5a374
> >> [0x00007f4209a59800+0x0000000000000b74]
> >> J 798 c1
> >>
> java.lang.invoke.MethodType$ConcurrentWeakInternSet.add(Ljava/lang/Object;)Ljava/lang/Object;
>
> >> java.base at 16-internal (66 bytes) @ 0x00007f42094af3b4
> >> [0x00007f42094af0a0+0x0000000000000314]
> >> J 880 c1
> >>
> java.lang.invoke.MethodType.makeImpl(Ljava/lang/Class;[Ljava/lang/Class;Z)Ljava/lang/invoke/MethodType;
>
> >> java.base at 16-internal (108 bytes) @ 0x00007f42094e2bcc
> >> [0x00007f42094e2720+0x00000000000004ac]
> >> J 1001 c1
> >>
> java.lang.invoke.MethodHandleNatives.findMethodHandleType(Ljava/lang/Class;[Ljava/lang/Class;)Ljava/lang/invoke/MethodType;
>
> >> java.base at 16-internal (7 bytes) @ 0x00007f420955997c
> >> [0x00007f4209559940+0x000000000000003c]
> >> v  ~StubRoutines::call_stub
> >> V  [libjvm.so+0xd0020a]  JavaCalls::call_helper(JavaValue*,
> >> methodHandle const&, JavaCallArguments*, Thread*)+0x62a
> >> V  [libjvm.so+0xd017e2]  JavaCalls::call_static(JavaValue*, Klass*,
> >> Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x312
> >> V  [libjvm.so+0x166c2e1]
> >> SystemDictionary::find_method_handle_type(Symbol*, Klass*,
> >> Thread*)+0x1541
> >> V  [libjvm.so+0x166e31f]
> >> SystemDictionary::find_method_handle_invoker(Klass*, Symbol*,
> >> Symbol*, Klass*, Handle*, Thread*)+0x5f
> >> V  [libjvm.so+0x112a15b]
> >> LinkResolver::lookup_polymorphic_method(LinkInfo const&, Handle*,
> >> Thread*)+0x52b
> >> V  [libjvm.so+0x112a6a7] LinkResolver::resolve_handle_call(CallInfo&,
> >> LinkInfo const&, Thread*)+0xa7
> >> V  [libjvm.so+0x112a944]
> >> LinkResolver::resolve_invokehandle(CallInfo&, constantPoolHandle
> >> const&, int, Thread*)+0x184
> >> V  [libjvm.so+0x11376c9] LinkResolver::resolve_invoke(CallInfo&,
> >> Handle, constantPoolHandle const&, int, Bytecodes::Code, Thread*)+0x69
> >> V  [libjvm.so+0x727eec]  Runtime1::patch_code(JavaThread*,
> >> Runtime1::StubID)+0x156c
> >> V  [libjvm.so+0x72a847]
> >> Runtime1::move_appendix_patching(JavaThread*)+0x37
> >> v  ~RuntimeStub::load_appendix_patching Runtime1 stub
> >> J 5283 c1 java.lang.ProcessHandleImpl$1.run()V java.base at 16-internal
> >> (136 bytes) @ 0x00007f420946c262 [0x00007f420946b3c0+0x0000000000000ea2]
> >> J 5275 c1
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
>
> >> java.base at 16-internal (187 bytes) @ 0x00007f420946d91c
> >> [0x00007f420946d680+0x000000000000029c]
> >> J 5272 c1 java.util.concurrent.ThreadPoolExecutor$Worker.run()V
> >> java.base at 16-internal (9 bytes) @ 0x00007f4209916c7c
> >> [0x00007f4209916be0+0x000000000000009c]
> >> J 3308 c1 java.lang.Thread.run()V java.base at 16-internal (17 bytes) @
> >> 0x00007f42097c2bf4 [0x00007f42097c2b60+0x0000000000000094]
> >> v  ~StubRoutines::call_stub
> >> V  [libjvm.so+0xd0020a]  JavaCalls::call_helper(JavaValue*,
> >> methodHandle const&, JavaCallArguments*, Thread*)+0x62a
> >> V  [libjvm.so+0xd00b15]  JavaCalls::call_virtual(JavaValue*, Klass*,
> >> Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x4b5
> >> V  [libjvm.so+0xd00fc1]  JavaCalls::call_virtual(JavaValue*, Handle,
> >> Klass*, Symbol*, Symbol*, Thread*)+0xb1
> >> V  [libjvm.so+0xe7ed63]  thread_entry(JavaThread*, Thread*)+0x123
> >> V  [libjvm.so+0x16d3f9c]  JavaThread::thread_main_inner()+0x21c
> >> V  [libjvm.so+0x16d9d20]  Thread::call_run()+0x100
> >> V  [libjvm.so+0x13ddbe6]  thread_native_entry(Thread*)+0x116
> >>
> >>
> >>
> >>
> >> On 7/22/20 9:16 PM, David Holmes wrote:
> >>> Hi Roger,
> >>>
> >>> On 23/07/2020 12:51 am, Roger Riggs wrote:
> >>>> Please review a change to the Process reaper thread initialization
> >>>> to preemptively
> >>>> make sure classes ThreadLocalRandom and ConcurrentHashMap are
> >>>> initialized.
> >>>
> >>> I don't see ThreadLocalRandom  appearing in any of the stack traces. ??
> >>>
> >>> David
> >>> -----
> >>>
> >>>>  From the stack overflow failures, it seems that the classes have
> >>>> not been initialized
> >>>> before they are used during processing the termination of a process.
> >>>> When the initialization is performed on the smaller reaper stack,
> >>>> it occasionally
> >>>> exceeds the available stack.
> >>>>
> >>>> As an aid to diagnostics,
> >>>> -XX:AbortVMOnException=java.lang.StackOverflowError
> >>>> is added to TestHumongousNonArrayAllocation that has failed
> >>>> intermittently.
> >>>> If the problem happens again it will produce an hs_error file with
> >>>> useful details
> >>>> and will otherwise not change the test behavior.
> >>>>
> >>>> Webrev:
> >>>> http://cr.openjdk.java.net/~rriggs/webrev-stackoverflow-8249217/
> >>>>
> >>>> Issue:
> >>>>    https://bugs.openjdk.java.net/browse/JDK-8249217
> >>>>
> >>>> Thanks, Roger
> >>>>
> >>
>
>