JDK-8231460: Java update from 11.0.11 to 11.0.13 changes JVM code cache behavior and results in more process cpu usage and unexpected profiled nmethods memory usage

[Schmidt, Lutz]

Hi Josef, Tobias, 

sorry for responding with some delay. Busy with other things I did not closely monitor all the OpenJDK mail traffic. 

The CodeCache behavior you describe sounds interesting. However, I do not see a way how the mentioned changes (JDK-8223444 and JDK-8231460) would cause or have any effect on that behavior. 

Why? The changes only modify the inner workings of the code cache. They have no effect on the contents (# of methods, method size, method lifetime, ...) of the code cache. Just the internal (free) space management is made more efficient and effective. 

To gain some insight into what's going on, you may want to use CodeHeap Analytics. The documentation for this feature can be found at 
  https://docs.oracle.com/en/java/javase/11/tools/java.html#GUID-4856361B-8BFD-4964-AE84-121F5F6CF111
Klick the download link to access the full documentation. 

To start with, 
  jcmd <pid> Compiler.CodeHeap_Analytics aggregate
will provide some general state information. I would suggest to collect the information in regular intervals during startup, until you see the drop in 'profiled nmethods' space. Please issue
  jcmd 23923 Compiler.CodeHeap_Analytics discard
at the end to free up JVM memory. 

Hope that helps! 

Best Regards,
Lutz


On 14.12.21, 14:31, "Tobias Hartmann" <tobias.hartmann at oracle.com> wrote:

    Hi Josef,

    Thanks for reporting this issue!

    Any chance you could run your application with different builds to narrow the problem down to a
    single change?

    @Lutz: As the author of 8223444 and 8231460, any ideas?

    Best regards,
    Tobias


    On 01.12.21 16:35, Josef Lehner wrote:
    > Dear OpenJDK team,
    > 
    > as described in this StackOverflow question, I want to reach out to you and
    > question whether the JVM code cache / codeheap still works as designed.
    > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstackoverflow.com%2Fquestions%2F70086548%2Fjava-update-from-11-0-11-to-11-0-13-changes-jvm-code-cache-behavior-and-results&data=04%7C01%7Clutz.schmidt%40sap.com%7C42d5016b11854667a90c08d9bf061369%7C42f7676cf455423c82f6dc2d99791af7%7C0%7C0%7C637750855085868849%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=8C59GLHHS%2FUSMWKHNJ23xZAD%2BskU0i2mmuWGPPrLU7E%3D&reserved=0
    > 
    > What we experience in our huge application is that with Java 11.0.13 and
    > -XX:ReservedCodeCacheSize=375m the code cache / codeheap 'profiled
    > nmethods' (C1 optimized, ~ 180 MB) drops after a very short time to a very
    > low level (less than 50 MB) and stays at this level forever while
    > 'non-profiled nmethods' (C2 optimized) is already at its limits.
    > After we tripled the -XX:ReservedCodeCacheSize to 1024 MB, both areas
    > 'profiled nmethods' and 'non-profiled nmethods' have stayed at a much
    > higher constant level (~ 258 MB) for over a week now instead of dropping to
    > less than 50 MB after 15 min (-XX:ReservedCodeCacheSize=375m) or dropping
    > after 3 hours (-XX:ReservedCodeCacheSize=512m).
    > From my point of view as a non-expert I would expect that the C1 optimized
    > code does not get removed (or at least not so much) from 'profiled
    > nmethods' as there is no space left in 'non-profiled nmethods' to optimize
    > it further. What do you think?
    > 
    > Important changes in 11.0.12:
    > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.openjdk.java.net%2Fbrowse%2FJDK-8223444&data=04%7C01%7Clutz.schmidt%40sap.com%7C42d5016b11854667a90c08d9bf061369%7C42f7676cf455423c82f6dc2d99791af7%7C0%7C0%7C637750855085868849%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Hm3EIcqksTv55T36l3GlgsEto1jctXcJNEJzZ65nlfY%3D&reserved=0 Improve CodeHeap Free
    > Space Management
    > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.openjdk.java.net%2Fbrowse%2FJDK-8231460&data=04%7C01%7Clutz.schmidt%40sap.com%7C42d5016b11854667a90c08d9bf061369%7C42f7676cf455423c82f6dc2d99791af7%7C0%7C0%7C637750855085868849%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=BBbrm3kqM9xSH%2FX7BwovduWR3C67ss0CgOL1YRooUOU%3D&reserved=0 Performance issue
    > (CodeHeap) with large free blocks
    > 
    > Best regards
    > Josef Lehner
    >