RFD: Grouping hot code in CodeCache

[Astigeevich, Evgeny]

Hi Dean,

I currently don’t know what hardware issues sparse code causes on Intel. I need to check which hardware counters get worse. It might be far vs near.

Graviton has a counter which measures issues with code placement and can be used to measure code sparsity. I think this counter is not connected to the issue of far calls vs near call. I can run experiments when we use far calls and near calls for dense code.

Thanks,
Evgeny

From: hotspot-dev <hotspot-dev-retn at openjdk.org> on behalf of "dean.long at oracle.com" <dean.long at oracle.com>
Date: Wednesday 5 March 2025 at 22:24
To: "hotspot-dev at openjdk.org" <hotspot-dev at openjdk.org>
Subject: RE: [EXTERNAL] RFD: Grouping hot code in CodeCache


CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



Just to clarify, if grouping helps, does that mean the reason for the performance impact of sparse code is mainly due to far calls vs near calls?

dl
On 3/5/25 10:41 AM, Astigeevich, Evgeny wrote:
Hi Vladimir,

This is JDK-8326205: Implement grouping hot nmethods in CodeCache.

As I managed to synthesize a benchmark (https://github.com/openjdk/jdk/pull/23831) to demonstrate performance impact of sparse code, I’d like to discuss a possible solution of the sparse code.

High level, a solution is:

1.       Detect hot code.

2.       Group hot code.

3.       Maintain grouped code.

Downstream we tried two approaches:
1.       Static lists of methods (compile command): Identify frequently used (hot) methods using test runs and provide static method lists to JVM in production. When JVM compiles a Java method and the method is on the list, JVM puts the code into to a designated code heap (HotCodeHeap).
2.       Dynamic lists of methods (compiler directives): Profile an application in production and dynamically relocate identified hot methods to HotCodeHeap. Relocation was implemented with recompilation.

The main advantage of static lists is zero profiling overhead in production. We do all profiling and analysis in test runs. Its problems are:

1.       Training Run Accuracy: We need training runs to have execution paths closely mimicking production environments. Otherwise we put wrong methods into HotCodeHeap.

2.       Method List Maintenance: We need to rerun training to regenerate lists when application code changes. Training runs are expensive and time-consuming. They require long runs to guarantee we see all major execution paths. Updating lists in production can be as complex as application deployment

3.       Method Placement Limitations: Methods marked for HotCodeHeap are permanently placed into HotCodeHeap. No mechanism to remove methods that become less frequently used.

We addressed these problems with dynamic lists of methods. We implemented a Java agent that runs within the same JVM to dynamically detect and manage hot Java methods without prior method identification. The agent detects hot methods using JFR. The agent manages hot Java methods in HotCodeHeap with compiler directives. A new compiler directive marks methods with dynamic states ("hot" or "cold"). Methods marked by the “hot” state are recompiled and placed in HotCodeHeap. Methods marked by the “cold” state are eventually removed from HotCodeHeap.

Problems of this approach are:

1.       It requires specific, complex modifications to compiler directive support: recompilation of Java methods affected by compiler directives changes. This functionality is unique to Java agent implementation and has limited potential for broader use.

2.       The agent cannot guarantee Java methods are moved to/removed from the HotCodeHeap because updates of compiler directives can fail.

3.       The agent knows nothing about compiled code, e.g. whether it’s C1 or C2 compiled, code size, profile. This data can useful for deciding to move or not to move to HotCodeHeap.

4.       Recompilations, especially C2, are expensive. Having many of them can cause performance issues. Also recompiled code might differ from the code we have detected as “hot”.

Running these two approaches in production we learned:

1.       We detect 95% of actively used methods withing the first 30 minutes of an application run. This is with JFR profiling configured: 90 seconds session duration, sampling each 11 ms, 8 minutes between profiling sessions. We can find actively used methods faster if we reduce a pause between profiling sessions and sampling period. However it will increase the profiling overhead and affect application performance. With the current configuration, the profiling overhead is between 1% - 2%.

2.       A set of actively used methods gets into the steady state (no new methods added to, no methods removed from) within the first 60 minutes.

3.       Static lists, when created from runs close to production, have 80% - 90% methods always in use. This does not change over time.

4.       Predicting the size of HotCodeHeap is difficult, especially with dynamic lists.

We want to have grouping of hot method functionality as a part Hotspot JVM. We will group only C2 compiled methods. We can group JVMCI compiled methods, e.g. Graal, if needed. We need profiling precise enough to detect major Java methods. Low overhead is more important than precision.

We think we can have a solution which does not require a lot of code:

5.       Detect hot code: we can an implementation based on the Sweeper: https://github.com/openjdk/jdk17u/blob/master/src/hotspot/share/runtime/sweeper.hpp. We will use the handshakes mechanism, what the Sweeper used, to detect nmethods on the top of thread stacks.

6.       Group hot code: we have a draft PR https://github.com/openjdk/jdk/pull/23573. It implements relocation of nmethods within CodeCache.

7.       Maintain grouped code: we will add an additional code heap where hot nmethods will be relocated to.

What do you think about this approach? Are there other possible solutions?

Thanks,
Evgeny A.



Amazon Development Centre (London) Ltd.Registered in England and Wales with registration number 04543232 with its registered office at 1 Principal Place, Worship Street, London EC2A 2FA, United Kingdom.




Amazon Development Centre (London) Ltd. Registered in England and Wales with registration number 04543232 with its registered office at 1 Principal Place, Worship Street, London EC2A 2FA, United Kingdom.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/hotspot-dev/attachments/20250305/b2b34728/attachment-0001.htm>