[External] : RFD: Grouping hot code in CodeCache

Thu Mar 6 22:41:15 UTC 2025

Hi Evgeny,

My concern is that it will complicate VM existing code for not 
significant benefits in real production environment.

What improvements your experiments in real production runs shows? And 
which JDK version you used for that?

As you know most of nmethod's metadata is moved from CodeCache. And 
Boris Ulasevich will move the final part (relocation info) soon. After 
that the code will be a lot more compact in CodeCache. Code sparsity 
should be less issue then.

It would be nice if you redo your production experiments after that.

I understand that we can still have sparsity due to "warm" nmethods  and 
C1 compiled code mixed with "hot" C2 nmethods.  I think compilation 
policy has heuristic to detect "warm" method (time intervals between 
invocations). Can we simply use a separate CodeCache's segment for all 
C2 "hot" (we can specify frequency flag to determine what "hot" means) 
methods regardless when they are compiled. Then you don't need to create 
list or do anything special for them.  Most likely we will waste more 
space in CodeCache but it could be conditional under flag which you 
already proposed in separate segment RFE.

Thanks,
Vladimir K

On 3/5/25 10:41 AM, Astigeevich, Evgeny wrote:
> Hi Vladimir,
> 
> This is JDK-8326205: Implement grouping hot nmethods in CodeCache.
 > > As I managed to synthesize a benchmark 
(https://github.com/openjdk/jdk/
> pull/23831 <https://urldefense.com/v3/__https://github.com/openjdk/jdk/ 
> pull/23831__;!!ACWV5N9M2RV99hQ!OwHez5zoUshzI- 
> baNlMChYzivbqU97PyvY08f_b1wH7Vd1hrqnwarTHE0Ha9IwOIOFw9jwE6gthfb- 
> imnfmmpfw$>) to demonstrate performance impact of sparse code, I’d like 
> to discuss a possible solution of the sparse code.
> 
> High level, a solution is:
> 
>   * Detect hot code.
>   * Group hot code.
>   * Maintain grouped code.
> 
> Downstream we tried two approaches:
> 
>   * *Static lists of methods (compile command):* Identify frequently
>     used (hot) methods using test runs and provide static method lists
>     to JVM in production. When JVM compiles a Java method and the method
>     is on the list, JVM puts the code into to a designated code heap
>     (HotCodeHeap).
>   * *Dynamic lists of methods (compiler directives):* Profile an
>     application in production and dynamically relocate identified hot
>     methods to HotCodeHeap. Relocation was implemented with recompilation.
> 
> The main advantage of static lists is zero profiling overhead in 
> production. We do all profiling and analysis in test runs. Its problems are:
> 
>   * *Training Run Accuracy*: We need training runs to have execution
>     paths closely mimicking production environments. Otherwise we put
>     wrong methods into HotCodeHeap.
>   * *Method List Maintenance:* We need to rerun training to regenerate
>     lists when application code changes. Training runs are expensive and
>     time-consuming. They require long runs to guarantee we see all major
>     execution paths. Updating lists in production can be as complex as
>     application deployment
>   * *Method Placement Limitations:* Methods marked for HotCodeHeap are
>     permanently placed into HotCodeHeap. No mechanism to remove methods
>     that become less frequently used.
> 
> We addressed these problems with dynamic lists of methods. We 
> implemented a Java agent that runs within the same JVM to dynamically 
> detect and manage hot Java methods without prior method identification. 
> The agent detects hot methods using JFR. The agent manages hot Java 
> methods in HotCodeHeap with compiler directives. A new compiler 
> directive marks methods with dynamic states ("hot" or "cold"). Methods 
> marked by the “hot” state are recompiled and placed in HotCodeHeap. 
> Methods marked by the “cold” state are eventually removed from HotCodeHeap.
> 
> Problems of this approach are:
> 
>   * It requires specific, complex modifications to compiler directive
>     support: recompilation of Java methods affected by compiler
>     directives changes. This functionality is unique to Java agent
>     implementation and has limited potential for broader use.
>   * The agent cannot guarantee Java methods are moved to/removed from
>     the HotCodeHeap because updates of compiler directives can fail.
>   * The agent knows nothing about compiled code, e.g. whether it’s C1 or
>     C2 compiled, code size, profile. This data can useful for deciding
>     to move or not to move to HotCodeHeap.
>   * Recompilations, especially C2, are expensive. Having many of them
>     can cause performance issues. Also recompiled code might differ from
>     the code we have detected as “hot”.
> 
> Running these two approaches in production we learned:
> 
>   * We detect 95% of actively used methods withing the first 30 minutes
>     of an application run. This is with JFR profiling configured: 90
>     seconds session duration, sampling each 11 ms, 8 minutes between
>     profiling sessions. We can find actively used methods faster if we
>     reduce a pause between profiling sessions and sampling period.
>     However it will increase the profiling overhead and affect
>     application performance. With the current configuration, the
>     profiling overhead is between 1% - 2%.
>   * A set of actively used methods gets into the steady state (no new
>     methods added to, no methods removed from) within the first 60 minutes.
>   * Static lists, when created from runs close to production, have 80% -
>     90% methods always in use. This does not change over time.
>   * Predicting the size of HotCodeHeap is difficult, especially with
>     dynamic lists.
> 
> We want to have grouping of hot method functionality as a part Hotspot 
> JVM. We will group only C2 compiled methods. We can group JVMCI compiled 
> methods, e.g. Graal, if needed. We need profiling precise enough to 
> detect major Java methods. Low overhead is more important than precision.
> 
> We think we can have a solution which does not require a lot of code:
> 
>   * Detect hot code: we can an implementation based on the Sweeper:
>     https://github.com/openjdk/jdk17u/blob/master/src/hotspot/share/
>     runtime/sweeper.hpp <https://urldefense.com/v3/__https://github.com/
>     openjdk/jdk17u/blob/master/src/hotspot/share/runtime/
>     sweeper.hpp__;!!ACWV5N9M2RV99hQ!OwHez5zoUshzI-
>     baNlMChYzivbqU97PyvY08f_b1wH7Vd1hrqnwarTHE0Ha9IwOIOFw9jwE6gthfb-
>     imVr_axpo$>. We will use the handshakes mechanism, what the Sweeper
>     used, to detect nmethods on the top of thread stacks.
>   * Group hot code: we have a draft PR https://github.com/openjdk/jdk/
>     pull/23573 <https://urldefense.com/v3/__https://github.com/openjdk/
>     jdk/pull/23573__;!!ACWV5N9M2RV99hQ!OwHez5zoUshzI-
>     baNlMChYzivbqU97PyvY08f_b1wH7Vd1hrqnwarTHE0Ha9IwOIOFw9jwE6gthfb-
>     imcL9xtiE$>. It implements relocation of nmethods within CodeCache.
>   * Maintain grouped code: we will add an additional code heap where hot
>     nmethods will be relocated to.
> 
> What do you think about this approach? Are there other possible solutions?
> 
> Thanks,
> 
> Evgeny A.
> 
> 
> 
> 
> Amazon Development Centre (London) Ltd.Registered in England and Wales 
> with registration number 04543232 with its registered office at 1 
> Principal Place, Worship Street, London EC2A 2FA, United Kingdom.
> 
> 