JDK 13 EA: 100% CPU use with no Java threads runnable

[Andrei Pangin]

Hi Stefan,

I think, async-profiler (
https://github.com/jvm-profiling-tools/async-profiler/) may give an idea
what's going on, since it collects the mixed profile including Java +
native + VM threads.
E.g. when the problem happens again, run the following command
./profiler.sh -d 20 -t -f profile.svg PID
(which means, collect CPU profile for 20 seconds in threaded mode and save
the output to profile.svg).

The last time I saw a similar issue, it was a C2 compiler thread falling
into an infinite loop (though that particular issue should have already
been fixed - https://bugs.openjdk.java.net/browse/JDK-8219448).

In this case, jstack or kill -3 PID should help, as the thread dump will
include compiler threads with the methods being compiled.

Regards,
Andrei

сб, 14 сент. 2019 г. в 22:16, Ioi Lam <ioi.lam at oracle.com>:

> Which version of gdb are you using? Maybe try upgrading gdb to a newer
> version? You might have to build it yourself, but that should be quite
> straight-forward on Linux.
>
>
> https://www.mapleprimes.com/questions/207040-Gdb-Crashes-When-Linking-To-OpenMaple-Library
>
> Another option is to send a kill signal to your process to create a core
> dump, and see if gdb can open the core file without crashing.
>
> See
>
> https://stackoverflow.com/questions/26727978/gdb-issue-bfd-reopening-tmp-no-such-file-or-directory/26767392#26767392
>
> Hopefully gdb won't do lazy loading when handling core files.
>
> Thanks
> - Ioi
>
>
> On 9/14/19 3:09 PM, Stefan Reich wrote:
> > gdb is segfaulting...
> >
> > > BFD: reopening
> > /home/stefan/.cache/JNA/temp/jna11400456213745436559.tmp: No such file
> > or directory
> > >
> > > BFD: reopening
> > /home/stefan/.cache/JNA/temp/jna11400456213745436559.tmp: No such file
> > or directory
> > >
> > > Can't read data for section '.eh_frame' in file
> > '/home/stefan/.cache/JNA/temp/jna11400456213745436559.tmp'
> >
> > I'm guessing the OSHI library is the trigger as it uses JNA. The same
> > module loads fine without gdb though.
> >
> >
> > On Sat, 14 Sep 2019 at 23:50, Stefan Reich
> > <stefan.reich.maker.of.eye at googlemail.com
> > <mailto:stefan.reich.maker.of.eye at googlemail.com>> wrote:
> >
> >     I will also try the gdb method... that one sounds like a very
> >     smart idea.
> >
> >     On Sat, 14 Sep 2019 at 23:49, Stefan Reich
> >     <stefan.reich.maker.of.eye at googlemail.com
> >     <mailto:stefan.reich.maker.of.eye at googlemail.com>> wrote:
> >
> >         Hi,
> >
> >         thanks for your answer. Unfortunately, I still don't know what
> >         triggers the "bad" state, so no test case... It happens
> >         randomly, but as I said, once it does, it doesn't go away
> >         anymore. (Still happening right now.)
> >
> >         Would the result from Thread.getAllStackTraces() help? I
> >         attached that to this email.
> >
> >         Many greetings,
> >         Stefan
> >
> >
> >         On Sat, 14 Sep 2019 at 23:43, Ioi Lam <ioi.lam at oracle.com
> >         <mailto:ioi.lam at oracle.com>> wrote:
> >
> >             Hi Stefan,
> >
> >             Is it possible for you to create a test case that
> >             reproduce this problem?
> >
> >             If not, try running the program inside a Linux terminal,
> >             and when the
> >             CPU goes at 100%, type "Ctrl-\" (Control + Backslash). It
> >             lists the
> >             states of all the threads that the JVM knows about. It
> >             might give you
> >             some idea what's happening.
> >
> >             An alternative is to run your program inside gdb,
> >
> >             $ gdb jdk13/bin/java
> >             (gdb) handle SIGSEGV noprint
> >             (gdb) handle SIGSEGV nostop
> >             (gdb) set print thread-events off
> >             (gdb) set height 0
> >             (gdb) run -cp myapp.jar App
> >
> >             (When the program enters the 100% CPU state, type Ctrl-C)
> >
> >             ^C
> >             Thread 1 "java" received signal SIGINT, Interrupt.
> >             0x00007ffff779798d in pthread_join (threadid=140737353910016,
> >             thread_return=0x7fffffff8508) at pthread_join.c:90
> >             90    pthread_join.c: No such file or directory.
> >             (gdb) thread apply all where
> >
> >             This will list all stacks of all the threads, including
> >             ones that the
> >             JVM may not be aware of. This should tell you who is
> >             spinning the CPU.
> >
> >             Thanks
> >             - Ioi
> >
> >
> >
> >
> >             On 9/14/19 1:57 PM, Stefan Reich wrote:
> >             > Hi,
> >             >
> >             > I was advised by Oracle's Fairoz Matte to contact this
> list.
> >             >
> >             > I am seeing a Java process use 100% of one CPU core
> >             continuously, but no
> >             > Java threads are runnable (as evidenced by internal
> >             thread sampling as well
> >             > as VisualVM).
> >             >
> >             > Whenever it occurs, the problem does not seem to go away
> >             until the Java VM
> >             > exits.
> >             >
> >             > I think I am using a single native library (OSHI), the
> >             rest is pure Java.
> >             >
> >             > Older versions of the JDK have exhibited the same
> >             problem, but most
> >             > interestingly, the bug is happening right now on a Linux
> >             machine
> >             > (Peppermint 7 I think) with OpenJDK 13 EA Build 31.
> >             >
> >             > I can perform any test on this machine that you instruct
> >             me to.
> >             >
> >             > Please advise,
> >             > Stefan
> >             >
> >
> >
> >
> >         --
> >         Stefan Reich
> >         BotCompany.de // Java-based operating systems
> >
> >
> >
> >     --
> >     Stefan Reich
> >     BotCompany.de // Java-based operating systems
> >
> >
> >
> > --
> > Stefan Reich
> > BotCompany.de // Java-based operating systems
>
>