CompileBroker hanging

[Gustav Åkesson]

Hello folks,

We a very peculiar issue that just started to happen in one of our
application instances, in which the JIT compiler was suddenly starting to
consume a lot of CPU (JVM had been running for a couple of weeks), and the
application was more sluggish as well. Even after removing the load towards
the instance the JIT compiler continued to consume CPU, for 5 more minutes.
Then it stopped. But when loading again the JIT also started to consume CPU
again.

Does anyone recognize this type of issue or is it a known issue?
I currently do not dare to restart the application since I then suspect the
issue will vanish and have no way of analyzing further. That limits my
options...

Here is two flamegraph pictures when the VM is (more or less) idle:
[image: fg_idle_overfiew.png]

[image: fg_idle_compilebroker.png]

And in a flamegraph with load I noticed that more time was spent in
interpreted mode as well.
It might be worth pointing out that when this issue happens the code cache
utilization is 115mb out 128mb (our reserved). But I also noticed that the
high-water mark was 127.1mb.
Is this a clue why the compiler is erratic? I'm thinking of bumping the
reserved code cache to 256mb as well.

This is what we're running on:
The VM it is running on RHEL 7, using 8 CPUs and 12GB of RAM.

*java version "1.8.0_231"*
Java(TM) SE Runtime Environment (build 1.8.0_231-b32)
Java HotSpot(TM) 64-Bit Server VM (build 25.231-b32, mixed mode)

*jcmd VM.flags:*
-XX:+AlwaysPreTouch
-XX:CICompilerCount=4
-XX:+CMSEdenChunksRecordAlways
-XX:CMSInitiatingOccupancyFraction=80
-XX:+CMSParallelInitialMarkEnabled
-XX:+CMSScavengeBeforeRemark
-XX:CMSWaitDuration=60000
-XX:CompressedClassSpaceSize=33554432
-XX:+DebugNonSafepoints
-XX:+DisableExplicitGC
-XX:ErrorFile=<error-file>
-XX:GCLogFileSize=31457280
-XX:InitialHeapSize=6215958528
-XX:MaxHeapSize=6215958528
-XX:MaxMetaspaceSize=268435456
-XX:MaxNewSize=1034944512
-XX:MaxTenuringThreshold=6
-XX:MetaspaceSize=268435456
-XX:MinHeapDeltaBytes=196608
-XX:NewSize=1034944512
-XX:NumberOfGCLogFiles=3
-XX:OldPLABSize=16
-XX:OldSize=5181014016
-XX:ParGCCardsPerStrideChunk=2048
-XX:+PreserveFramePointer
-XX:+PrintGC
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintTenuringDistribution
*-XX:ReservedCodeCacheSize=134217728 *
*-XX:+TieredCompilation*
-XX:+UnlockDiagnosticVMOptions
-XX:+UseBiasedLocking
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseCompressedClassPointers
-XX:+UseCompressedOops
-XX:+UseConcMarkSweepGC
-XX:+UseFastUnorderedTimeStamps
-XX:+UseGCLogFileRotation
-XX:+UseParNewGC


Best Regards,
Gustav Åkesson