RFR: Save/load nmethod without going through CodeBuffer [v2]

[Ashutosh Mehra]

On Mon, 17 Mar 2025 11:53:32 GMT, Ashutosh Mehra <asmehra at openjdk.org> wrote:

>> This is the prototype for storing and loading nmethods without going through the CodeBuffer.
>> The new implementation is protected by the flag -XX:+UseNewCode2.
>> 
>> Some numbers using this implementation:
>> spring-boot-getting-started [0] shows startup improvement of ~ 7.5%% and quarkus-getting-started [1] shows improvement of around ~3.5%.
>> 
>> Numbers for Springboot
>> 
>> Old build = /home/asmehra/data/ashu-mehra/leyden/build/nmethod-single-copy-load-release/images/jdk with options -XX:+UnlockDiagnosticVMOptions -XX:-UseNewCode2
>> New build = /home/asmehra/data/ashu-mehra/leyden/build/nmethod-single-copy-load-release/images/jdk with options -XX:+UnlockDiagnosticVMOptions -XX:+UseNewCode2
>> Run,Old CDS + AOT,New CDS + AOT
>> 1,544,512
>> 2,554,525
>> 3,550,508
>> 4,571,515
>> 5,550,506
>> 6,552,515
>> 7,568,510
>> 8,554,517
>> 9,551,506
>> 10,556,508
>> Geomean,554.94,512.17
>> Stdev,7.90,5.65
>> 
>> Numbers for Quarkus:
>> 
>> Old build = /home/asmehra/data/ashu-mehra/leyden/build/nmethod-single-copy-load-release/images/jdk with options -XX:+UnlockDiagnosticVMOptions -XX:-UseNewCode2
>> New build = /home/asmehra/data/ashu-mehra/leyden/build/nmethod-single-copy-load-release/images/jdk with options -XX:+UnlockDiagnosticVMOptions -XX:+UseNewCode2
>> Run,Old CDS + AOT,New CDS + AOT
>> 1,359,346
>> 2,360,353
>> 3,373,357
>> 4,376,356
>> 5,366,353
>> 6,360,356
>> 7,361,361
>> 8,347,349
>> 9,355,336
>> 10,374,342
>> Geomean,363.00,350.82
>> Stdev,8.70,7.27
>> 
>> 
>> -Xlog:init logs the load time from AOT code cache at JVM exit.
>> For spring-boot-getting-started without `UseNewCode2`:
>> 
>> [3.459s][info][init]     SC Load Time:           0.202 s
>> [3.459s][info][init]       nmethod register:       0.135 s
>> [3.459s][info][init]       find cached code:       0.007 s
>> 
>> 
>> For spring-boot-getting-started with `UseNewCode2`:
>> 
>> [3.192s][info][init] 
>> [3.192s][info][init]     SC Load Time:           0.138 s
>> [3.192s][info][init]       nmethod register:       0.111 s
>> [3.192s][info][init]       find cached code:       0.006 s
>> 
>> 
>> For quarkus-getting-started without `UseNewCode2`
>> 
>> [0.392s][info][init]     SC Load Time:           0.060 s
>> [0.392s][info][init]       nmethod register:       0.039 s
>> [0.392s][info][init]       find cached code:       0.002 s
>> 
>> 
>> For quarkus-getting-started with `UseNewCode2`
>> 
>> [0.386s][info][init]     SC Load Time:           0.033 s
>> [0.386s][info][init]       nmethod register:       0.027 s
>> [0.386s][info][init]       find cached code:      ...
>
> Ashutosh Mehra has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Minor cleanup
>   
>   Signed-off-by: Ashutosh Mehra <asmehra at redhat.com>

I have updated this PR to remove `SCnmethod` and store `nmethod` directly into AOT code cache.
Perf numbers have changed compared to the last prototype.
For spring boot:

Old build = /home/asmehra/data/ashu-mehra/leyden/build/premain-release/images/jdk with options
New build = /home/asmehra/data/ashu-mehra/leyden/build/nmethod-single-copy-load-release/images/jdk with options
Run,Old CDS + AOT,New CDS + AOT
1,544,532
2,548,526
3,558,534
4,550,529
5,541,532
6,540,540
7,539,542
8,554,526
9,540,532
10,544,530
Geomean,545.77,532.28
Stdev,6.18,5.02


For quarkus:

Old build = /home/asmehra/data/ashu-mehra/leyden/build/premain-release/images/jdk with options
New build = /home/asmehra/data/ashu-mehra/leyden/build/nmethod-single-copy-load-release/images/jdk with options
Run,Old CDS + AOT,New CDS + AOT
1,358,349
2,355,356
3,360,347
4,372,342
5,357,353
6,344,336
7,356,353
8,352,339
9,359,338
10,347,340
Geomean,355.93,345.23
Stdev,7.27,6.84


This shows an improvement of 2.5~3%.

`SC Load Time` as reported by `-Xlog:init` shows it is cut down to almost half.
Springboot without this change:

[0.810s][info][init]     SC Load Time:           0.127 s
[0.810s][info][init]       nmethod register:       0.091 s
[0.810s][info][init]       find cached code:       0.003 s


Springboot with this change:

[0.800s][info][init]     SC Load Time:           0.064 s
[0.800s][info][init]       nmethod register:       0.055 s
[0.800s][info][init]       find cached code:       0.003 s


Quarkus without this change:

[0.392s][info][init]     SC Load Time:           0.056 s
[0.392s][info][init]       nmethod register:       0.040 s
[0.392s][info][init]       find cached code:       0.002 s

Quarkus with this change:

[0.383s][info][init]     SC Load Time:           0.027 s
[0.383s][info][init]       nmethod register:       0.024 s
[0.383s][info][init]       find cached code:       0.002 s

-------------

PR Comment: https://git.openjdk.org/leyden/pull/27#issuecomment-2729228757