CompileBroker hanging

[Nils Eliasson]

Hi Gustav,

It sounds like you run out of code cache. When the code cache is getting 
full a number of things might happen (1) The code cache is swept more 
often which consumes CPU time (2) The oldest compiled methods that 
hasn't been used recently will be evicted. Throwing out the oldest 
compiled methods is a last effort to make room. if the compiled code 
that is evicted is used eventually, it will trigger new compilations. If 
all the code is fairly hot - the VM will be alternate between compiling 
and throwing out code.

You can turn off code cache flushing with -XX:-UseCodeCacheFlushing but 
then when the code cache is full - no more compilations will happen, and 
you might end up spending a lot of time in the interpreter.

 From your numbers, your code cache is pretty full - there is always 
some amount of fragmentation that is reported as free. I suggest you 
increase the code cache size. You write that the server has been running 
for a couple of weeks - that suggest that you only need a modest increase.

When you upgrade to a newer JDK there is both jcmds and JFR events that 
is useful for extracting code cache statistics. Since JDK 9 there is 
also the Segmented Code Cache that helps mitigate fragmentation in the 
code cache.

Regards,

Nils Eliasson

On 2020-03-25 08:14, Gustav Åkesson wrote:
> Hello folks,
>
> We a very peculiar issue that just started to happen in one of our
> application instances, in which the JIT compiler was suddenly starting to
> consume a lot of CPU (JVM had been running for a couple of weeks), and the
> application was more sluggish as well. Even after removing the load towards
> the instance the JIT compiler continued to consume CPU, for 5 more minutes.
> Then it stopped. But when loading again the JIT also started to consume CPU
> again.
>
> Does anyone recognize this type of issue or is it a known issue?
> I currently do not dare to restart the application since I then suspect the
> issue will vanish and have no way of analyzing further. That limits my
> options...
>
> Here is two flamegraph pictures when the VM is (more or less) idle:
> [image: fg_idle_overfiew.png]
>
> [image: fg_idle_compilebroker.png]
>
> And in a flamegraph with load I noticed that more time was spent in
> interpreted mode as well.
> It might be worth pointing out that when this issue happens the code cache
> utilization is 115mb out 128mb (our reserved). But I also noticed that the
> high-water mark was 127.1mb.
> Is this a clue why the compiler is erratic? I'm thinking of bumping the
> reserved code cache to 256mb as well.
>
> This is what we're running on:
> The VM it is running on RHEL 7, using 8 CPUs and 12GB of RAM.
>
> *java version "1.8.0_231"*
> Java(TM) SE Runtime Environment (build 1.8.0_231-b32)
> Java HotSpot(TM) 64-Bit Server VM (build 25.231-b32, mixed mode)
>
> *jcmd VM.flags:*
> -XX:+AlwaysPreTouch
> -XX:CICompilerCount=4
> -XX:+CMSEdenChunksRecordAlways
> -XX:CMSInitiatingOccupancyFraction=80
> -XX:+CMSParallelInitialMarkEnabled
> -XX:+CMSScavengeBeforeRemark
> -XX:CMSWaitDuration=60000
> -XX:CompressedClassSpaceSize=33554432
> -XX:+DebugNonSafepoints
> -XX:+DisableExplicitGC
> -XX:ErrorFile=<error-file>
> -XX:GCLogFileSize=31457280
> -XX:InitialHeapSize=6215958528
> -XX:MaxHeapSize=6215958528
> -XX:MaxMetaspaceSize=268435456
> -XX:MaxNewSize=1034944512
> -XX:MaxTenuringThreshold=6
> -XX:MetaspaceSize=268435456
> -XX:MinHeapDeltaBytes=196608
> -XX:NewSize=1034944512
> -XX:NumberOfGCLogFiles=3
> -XX:OldPLABSize=16
> -XX:OldSize=5181014016
> -XX:ParGCCardsPerStrideChunk=2048
> -XX:+PreserveFramePointer
> -XX:+PrintGC
> -XX:+PrintGCDateStamps
> -XX:+PrintGCDetails
> -XX:+PrintGCTimeStamps
> -XX:+PrintTenuringDistribution
> *-XX:ReservedCodeCacheSize=134217728 *
> *-XX:+TieredCompilation*
> -XX:+UnlockDiagnosticVMOptions
> -XX:+UseBiasedLocking
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseCompressedClassPointers
> -XX:+UseCompressedOops
> -XX:+UseConcMarkSweepGC
> -XX:+UseFastUnorderedTimeStamps
> -XX:+UseGCLogFileRotation
> -XX:+UseParNewGC
>
>
> Best Regards,
> Gustav Åkesson