CompileBroker hanging

[Vladimir Ivanov]

Hi Gustav,

Unfortunately, the attachments were stripped.

Can you, please, publish them somewhere and post the links here?

Based on your description, it look like a continuous recompilation 
caused by code cache overflow.

Best regards,
Vladimir Ivanov

On 25.03.2020 10:14, Gustav Åkesson wrote:
> Hello folks,
> 
> We a very peculiar issue that just started to happen in one of our
> application instances, in which the JIT compiler was suddenly starting to
> consume a lot of CPU (JVM had been running for a couple of weeks), and the
> application was more sluggish as well. Even after removing the load towards
> the instance the JIT compiler continued to consume CPU, for 5 more minutes.
> Then it stopped. But when loading again the JIT also started to consume CPU
> again.
> 
> Does anyone recognize this type of issue or is it a known issue?
> I currently do not dare to restart the application since I then suspect the
> issue will vanish and have no way of analyzing further. That limits my
> options...
> 
> Here is two flamegraph pictures when the VM is (more or less) idle:
> [image: fg_idle_overfiew.png]
> 
> [image: fg_idle_compilebroker.png]
> 
> And in a flamegraph with load I noticed that more time was spent in
> interpreted mode as well.
> It might be worth pointing out that when this issue happens the code cache
> utilization is 115mb out 128mb (our reserved). But I also noticed that the
> high-water mark was 127.1mb.
> Is this a clue why the compiler is erratic? I'm thinking of bumping the
> reserved code cache to 256mb as well.
> 
> This is what we're running on:
> The VM it is running on RHEL 7, using 8 CPUs and 12GB of RAM.
> 
> *java version "1.8.0_231"*
> Java(TM) SE Runtime Environment (build 1.8.0_231-b32)
> Java HotSpot(TM) 64-Bit Server VM (build 25.231-b32, mixed mode)
> 
> *jcmd VM.flags:*
> -XX:+AlwaysPreTouch
> -XX:CICompilerCount=4
> -XX:+CMSEdenChunksRecordAlways
> -XX:CMSInitiatingOccupancyFraction=80
> -XX:+CMSParallelInitialMarkEnabled
> -XX:+CMSScavengeBeforeRemark
> -XX:CMSWaitDuration=60000
> -XX:CompressedClassSpaceSize=33554432
> -XX:+DebugNonSafepoints
> -XX:+DisableExplicitGC
> -XX:ErrorFile=<error-file>
> -XX:GCLogFileSize=31457280
> -XX:InitialHeapSize=6215958528
> -XX:MaxHeapSize=6215958528
> -XX:MaxMetaspaceSize=268435456
> -XX:MaxNewSize=1034944512
> -XX:MaxTenuringThreshold=6
> -XX:MetaspaceSize=268435456
> -XX:MinHeapDeltaBytes=196608
> -XX:NewSize=1034944512
> -XX:NumberOfGCLogFiles=3
> -XX:OldPLABSize=16
> -XX:OldSize=5181014016
> -XX:ParGCCardsPerStrideChunk=2048
> -XX:+PreserveFramePointer
> -XX:+PrintGC
> -XX:+PrintGCDateStamps
> -XX:+PrintGCDetails
> -XX:+PrintGCTimeStamps
> -XX:+PrintTenuringDistribution
> *-XX:ReservedCodeCacheSize=134217728 *
> *-XX:+TieredCompilation*
> -XX:+UnlockDiagnosticVMOptions
> -XX:+UseBiasedLocking
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseCompressedClassPointers
> -XX:+UseCompressedOops
> -XX:+UseConcMarkSweepGC
> -XX:+UseFastUnorderedTimeStamps
> -XX:+UseGCLogFileRotation
> -XX:+UseParNewGC
> 
> 
> Best Regards,
> Gustav Åkesson
>