JVM stuck/looping in futex call

[Sundara Mohan M]

glibc-2.12-1.212.el6.x86_64
compat-glibc-2.5-46.2.x86_64
glibc-utils-2.12-1.212.el6.x86_64
glibc-common-2.12-1.212.el6.x86_64
Linux 2.6.32-754.10.1.el6.20190116.16.x86_64 #1 SMP Wed Jan 16 21:27:59 UTC
2019 x86_64 x86_64 x86_64 GNU/Linux (RHEL-6.10)

Thanks
Sundar



On Tue, Nov 5, 2019 at 3:52 PM David Holmes <david.holmes at oracle.com> wrote:

> On 5/11/2019 4:13 pm, Sundara Mohan M wrote:
> > Hi David,
> >      Will try to get stack when it happens.
> > I think the main thread is where the loop happens (in my case it is
> > jetty server).
> > No special environment just jetty server and no native threads are
> > attached. We also try to avoid JNI related stuff as much as possible.
>
> Can you provide full details on OS kernel and glibc versions?
>
> No one else has reported anything like this.
>
> Thanks,
> David
>
> >
> > Thanks
> > Sundar
> >
> > On Mon, Nov 4, 2019 at 3:17 PM David Holmes <david.holmes at oracle.com
> > <mailto:david.holmes at oracle.com>> wrote:
> >
> >     On 5/11/2019 8:43 am, Sundara Mohan M wrote:
> >      > HI David,
> >      >      Did you mean to get stack trace of that process? I could
> >     attach to
> >      > gdb but not sure where to keep breakpoint.
> >      > More info on how to get this will be helpful.
> >
> >     I need to see the stack before we hit the looping call, to see what
> it
> >     is that triggers the loop. Can you tell what thread is involved?
> >
> >     Is there something special/different about your Linux environment? Do
> >     you have native threads attached to the VM?
> >
> >     Thanks,
> >     David
> >
> >      >
> >      > Thanks
> >      > Sundar
> >      >
> >      > On Fri, Nov 1, 2019 at 4:03 PM David Holmes
> >     <david.holmes at oracle.com <mailto:david.holmes at oracle.com>
> >      > <mailto:david.holmes at oracle.com
> >     <mailto:david.holmes at oracle.com>>> wrote:
> >      >
> >      >     Hi Sundar,
> >      >
> >      >     On 2/11/2019 5:39 am, Sundara Mohan M wrote:
> >      >      > Hi,
> >      >      >      I am running openjdk12/Linux on our systems and see
> >     jvm not
> >      >     responding
> >      >      > to jstack or any diagnostic command (jcmd
> >     VM.info/Thread.print).
> >      >     Though
> >      >      > application is running fine.
> >      >
> >      >     That would sound like the attach thread (which would respond
> >     to the
> >      >     jstack or other diagnostic command) is in some kind of bad
> state.
> >      >
> >      >      > I see following stack track
> >      >      >
> >      >      > Process 115586 attached
> >      >      > futex(0x7feeeffa19d0, FUTEX_WAIT, 116198, NULL) = ?
> >     ERESTARTSYS
> >      >     (To be
> >      >      > restarted if SA_RESTART is set)
> >      >      > --- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER,
> si_pid=115587,
> >      >     si_uid=1000}
> >      >      > ---
> >      >      > futex(0x7feee8008bf8, FUTEX_WAKE_PRIVATE, 1) = 1
> >      >      > rt_sigreturn()                          = 202
> >      >      > futex(0x7feeeffa19d0, FUTEX_WAIT, 116198, NULL) = ?
> >     ERESTARTSYS
> >      >     (To be
> >      >      > restarted if SA_RESTART is set)
> >      >      > --- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER,
> si_pid=115587,
> >      >     si_uid=1000}
> >      >      > ---
> >      >      > futex(0x7feee8008bf8, FUTEX_WAKE_PRIVATE, 1) = 1
> >      >      > rt_sigreturn()                          = 202
> >      >      > futex(0x7feeeffa19d0, FUTEX_WAIT, 116198, NULL) = ?
> >     ERESTARTSYS
> >      >     (To be
> >      >      > restarted if SA_RESTART is set)
> >      >      > --- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER,
> si_pid=115587,
> >      >     si_uid=1000}
> >      >      > ---
> >      >      > futex(0x7feee8008bf8, FUTEX_WAKE_PRIVATE, 1) = 1
> >      >      > rt_sigreturn()                          = 202
> >      >      > futex(0x7feeeffa19d0, FUTEX_WAIT, 116198, NULL) = ?
> >     ERESTARTSYS
> >      >     (To be
> >      >      > restarted if SA_RESTART is set)
> >      >      > --- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER,
> si_pid=115587,
> >      >     si_uid=1000}
> >      >      > ---
> >      >      > rt_sigreturn()                          = 202
> >      >      > futex(0x7feeeffa19d0, FUTEX_WAIT, 116198, NULL) = ?
> >     ERESTARTSYS
> >      >     (To be
> >      >      > restarted if SA_RESTART is set)
> >      >      > --- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER,
> si_pid=115587,
> >      >     si_uid=1000}
> >      >      > ---
> >      >      > rt_sigreturn()                          = 202
> >      >      > futex(0x7feeeffa19d0, FUTEX_WAIT, 116198, NULL^CProcess
> 115586
> >      >     detached
> >      >      >   <detached ...>
> >      >      >
> >      >      > Can someone help me understand what is happening here?
> >      >
> >      >     It appears that in responding to the SIGQUIT that is used to
> >     trigger
> >      >     the
> >      >     starting of the attach listener thread, that something is
> >     going wrong.
> >      >     We appear to be continually restarting an operation that
> >     still sees the
> >      >     signal pending - which doesn't really make sense to me. Can
> >     you get a
> >      >     complete stack trace using gdb?
> >      >
> >      >      > Please redirect me to proper ilist if this is not correct
> list
> >      >     for these
> >      >      > type of questions.
> >      >
> >      >     This list is fine. It may end up being an issue for
> >     serviceability-dev
> >      >     but we can deal with that later. :)
> >      >
> >      >     Thanks,
> >      >     David
> >      >     -----
> >      >
> >      >      >
> >      >      > TIA
> >      >      > Sundar
> >      >      >
> >      >
> >
>