JDK-8231460: Java update from 11.0.11 to 11.0.13 changes JVM code cache behavior and results in more process cpu usage and unexpected profiled nmethods memory usage

[Schmidt, Lutz]

Hi Josef, 

thank you for the log data you provided. I was offline during Christmas/New Year, so I had a chance to look at the data only today. 

Looking at the "aggregate" output at various timestamps, it is obvious that the CodeHeap for non-profiled nmethods is near its max capacity. That is true for both runs (11.0.11 and 11.0.13). The CodeHeap for profiled nmethods, on the other hand, shows plenty of free space (50% and 80%). The CodeHeap free space appears less fragmented with 11.0.13. That is most likely a result of the improvements introduced with JDK-8223444 and JDK-8231460. If at all, the reduced fragmentation would have a positive effect.

The nmethod sweep rate (over time) is another strong indicator that the CodeHeap is sized somewhat too small for the application. After warmup (timestamps 172747 and 172753 for 11.0.11; 173023 and 173030 for 11.0.13), the segment for non-profiled nmethods reaches max capacity (>98%). The sweeper flushes close to 10k C2-compiled methods within six seconds. 

Sweeper activity continues to be high throughout the rest of the sampling period, with 11.0.13 being significantly more active than 11.0.11. Higher sweeper activity indicates higher stress on the CodeHeap. I can only speculate on the triggering factors. It could be slightly different compilation patterns (compile time per method, sequence of compilations, variations in inlining decisions) and varying compiled method sizes due to changed speed/size tradeoffs in the JIT compiler.

The CodeHeap size seems to be tuned to keep the system just barely on the good side of the tipping point. In such a situation, minimal changes may cause dramatic effects. In my opinion, we see exactly that.

Aside from all speculation: what you don't want is permanent recompilation of nmethods. It slows down your application because the method is executed in interpreted mode until the compilation is ready. Furthermore, it consumes many cpu cycles for recompilation which could be used elsewhere (or could just be saved).

What's the remedy?
==================

As you already mentioned, there is easy relief if you increase the CodeHeap size to 1G. The resulting space for non-profiled nmethods (approx. 500M) is obviously enough to hold all compiled nmethods.

If that is not an option (too expensive), you could try to turn of CodeCache segmentation with -XX:-SegmentedCodeCache. With a little luck, your application may reach "steady state", i.e. only few recompilations after warmup, with -XX:ReservedCodeCacheSize=512m or even a bit less. 

Best Regards,
Lutz



On 29.12.21, 12:28, "hotspot-compiler-dev on behalf of Josef Lehner" <hotspot-compiler-dev-retn at openjdk.java.net on behalf of home.josef.lehner at gmail.com> wrote:

    Hi Tobias, Lutz,

    Thanks for your insights.

    I updated the Stackoverflow question (UPDATE 2021-12-28) with the requested
    data, running the jcmd command every 5 seconds: jcmd <pid>
    Compiler.CodeHeap_Analytics aggregate

    https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstackoverflow.com%2Fquestions%2F70086548%2Fjava-update-from-11-0-11-to-11-0-13-changes-jvm-code-cache-behavior-and-results&data=04%7C01%7Clutz.schmidt%40sap.com%7C272cdb4b39d14a8665d208d9cabe5171%7C42f7676cf455423c82f6dc2d99791af7%7C0%7C0%7C637763741036001724%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=NdNhBnJrjJilHwq0I5kCpKtXC13agVVb5EB0%2BycycHg%3D&reserved=0

    You can access the zip file with the output right here too:
    https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdrive.google.com%2Ffile%2Fd%2F1PLCo1XnKBHaGdGpVgzHraTy1pJxs4Id9%2Fview%3Fusp%3Dsharing&data=04%7C01%7Clutz.schmidt%40sap.com%7C272cdb4b39d14a8665d208d9cabe5171%7C42f7676cf455423c82f6dc2d99791af7%7C0%7C0%7C637763741036001724%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=QXYH8ogJcjk770AgHLY%2B0Jh6dqX1R74SQf6bLSFfmpE%3D&reserved=0

    Best regards and a happy new year,
    Josef