JDK-8231460: Java update from 11.0.11 to 11.0.13 changes JVM code cache behavior and results in more process cpu usage and unexpected profiled nmethods memory usage

[Schmidt, Lutz]

Some additional thoughts: 

Obviously, the code cache is sized too tight to reach a steady state. While the system is warming up, methods are first C1-compiled (with profiling). That fills the code cache segment for profiled nmethods. Over time, many of these methods become hotter and their C2-compilation is triggered, filling the segment for non-profiled nmethods. Once a C2-compiled copy is available, the related C1-compiled copy becomes obsolete and will be swept away over time. 

As time progresses, more and more of the Java code is considered (really) hot and gets C2-compiled. There is not much left in the C1-compiled state. That explains why you see a usage drop in the 'profiled nmethods' segment.

The 'non-profiled nmethods' segment, on the other hand, is packed full. You should see permanent recompilation events and permanent high code cache sweeper activity. Simply said, sizzling hot methods are C2-(re)compiled, pushing out methods which have cooled off a bit. That is the "good" case. If sweeping can't keep up, JIT compilation might as well get disabled altogether. Your application would then continue to run with that static state of the code cache. Because all methods are hot, they are not considered for recompilation by C1.

Those compiler gurus out there are welcome to correct me if I'm wrong!

Best Regards,
Lutz 


On 15.12.21, 10:27, "Schmidt, Lutz" <lutz.schmidt at sap.com> wrote:

    Hi Josef, Tobias, 

    sorry for responding with some delay. Busy with other things I did not closely monitor all the OpenJDK mail traffic. 

    The CodeCache behavior you describe sounds interesting. However, I do not see a way how the mentioned changes (JDK-8223444 and JDK-8231460) would cause or have any effect on that behavior. 

    Why? The changes only modify the inner workings of the code cache. They have no effect on the contents (# of methods, method size, method lifetime, ...) of the code cache. Just the internal (free) space management is made more efficient and effective. 

    To gain some insight into what's going on, you may want to use CodeHeap Analytics. The documentation for this feature can be found at 
      https://docs.oracle.com/en/java/javase/11/tools/java.html#GUID-4856361B-8BFD-4964-AE84-121F5F6CF111
    Klick the download link to access the full documentation. 

    To start with, 
      jcmd <pid> Compiler.CodeHeap_Analytics aggregate
    will provide some general state information. I would suggest to collect the information in regular intervals during startup, until you see the drop in 'profiled nmethods' space. Please issue
      jcmd 23923 Compiler.CodeHeap_Analytics discard
    at the end to free up JVM memory. 

    Hope that helps! 

    Best Regards,
    Lutz


    On 14.12.21, 14:31, "Tobias Hartmann" <tobias.hartmann at oracle.com> wrote:

        Hi Josef,

        Thanks for reporting this issue!

        Any chance you could run your application with different builds to narrow the problem down to a
        single change?

        @Lutz: As the author of 8223444 and 8231460, any ideas?

        Best regards,
        Tobias


        On 01.12.21 16:35, Josef Lehner wrote:
        > Dear OpenJDK team,
        > 
        > as described in this StackOverflow question, I want to reach out to you and
        > question whether the JVM code cache / codeheap still works as designed.
        > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstackoverflow.com%2Fquestions%2F70086548%2Fjava-update-from-11-0-11-to-11-0-13-changes-jvm-code-cache-behavior-and-results&data=04%7C01%7Clutz.schmidt%40sap.com%7C42d5016b11854667a90c08d9bf061369%7C42f7676cf455423c82f6dc2d99791af7%7C0%7C0%7C637750855085868849%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=8C59GLHHS%2FUSMWKHNJ23xZAD%2BskU0i2mmuWGPPrLU7E%3D&reserved=0
        > 
        > What we experience in our huge application is that with Java 11.0.13 and
        > -XX:ReservedCodeCacheSize=375m the code cache / codeheap 'profiled
        > nmethods' (C1 optimized, ~ 180 MB) drops after a very short time to a very
        > low level (less than 50 MB) and stays at this level forever while
        > 'non-profiled nmethods' (C2 optimized) is already at its limits.
        > After we tripled the -XX:ReservedCodeCacheSize to 1024 MB, both areas
        > 'profiled nmethods' and 'non-profiled nmethods' have stayed at a much
        > higher constant level (~ 258 MB) for over a week now instead of dropping to
        > less than 50 MB after 15 min (-XX:ReservedCodeCacheSize=375m) or dropping
        > after 3 hours (-XX:ReservedCodeCacheSize=512m).
        > From my point of view as a non-expert I would expect that the C1 optimized
        > code does not get removed (or at least not so much) from 'profiled
        > nmethods' as there is no space left in 'non-profiled nmethods' to optimize
        > it further. What do you think?
        > 
        > Important changes in 11.0.12:
        > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.openjdk.java.net%2Fbrowse%2FJDK-8223444&data=04%7C01%7Clutz.schmidt%40sap.com%7C42d5016b11854667a90c08d9bf061369%7C42f7676cf455423c82f6dc2d99791af7%7C0%7C0%7C637750855085868849%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Hm3EIcqksTv55T36l3GlgsEto1jctXcJNEJzZ65nlfY%3D&reserved=0 Improve CodeHeap Free
        > Space Management
        > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.openjdk.java.net%2Fbrowse%2FJDK-8231460&data=04%7C01%7Clutz.schmidt%40sap.com%7C42d5016b11854667a90c08d9bf061369%7C42f7676cf455423c82f6dc2d99791af7%7C0%7C0%7C637750855085868849%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=BBbrm3kqM9xSH%2FX7BwovduWR3C67ss0CgOL1YRooUOU%3D&reserved=0 Performance issue
        > (CodeHeap) with large free blocks
        > 
        > Best regards
        > Josef Lehner
        >