<div dir="ltr"><div>Hi again!</div><div><br></div><div>Comparing native and java was not as straight forward as I thought... but I decided to just do an experiment: What would happen if I train with "-Xcomp" and force compilation of everything? Would I get some advantage?</div><div><br></div><div>My hypothesis said yes. Reality had other ideas.</div><div><br></div><div>This is a simple REST API over Quarkus that calls a database and returns a
select. I trained with "-Xcomp" and then run production without that
option. And compared production runs with what happens if I don't train with XComp.</div><div>This was done on 2 cores with dedicated cpu cores. </div><div>But on my laptop, so other things running at the same time may have interfered (like using IO or memory, who knows, slack is a beast). But I run it four times and the results are always similar. </div><div><br></div><div>JDK26 said that "[warning][aot] The AOT cache was created by a different version or build of HotSpot" so I couldn't even use it on my experiment.</div><div>Premain (results from build over 127bfc9b0dd122c78e702867a88e0847ec362e68) didn't throw that error. Probably this is a bug, not a feature, but let's use it!</div><div><br></div><div>Do we store more stuff on the cache with that option enabled? Yes, we definitely do. </div><div><br></div><div><div class="gmail_chip gmail_chip_bubble gmail_spinner" style="width:0px;height:0px;border:1px dashed #e0e0e0;background-color:#f0f0f0;display:inline-block"></div><img src="cid:ii_ml6h7la44" alt="image.png" width="578" height="357"></div><div><br></div><div>Do we have a faster start-up time with Xcomp enabled? No, we even have a worse start-up time:</div><div><br></div><div><img src="cid:ii_ml6f0ddm2" alt="image.png" width="578" height="359"></div><div>I decided to take a look at the cache statistics in both Premain runs:</div><div><br></div><div>With XComp on:</div><div>[debug][aot,codecache,exit] Adapters: total=725<br>[debug][aot,codecache,exit] Shared Blobs: total=0<br>[debug][aot,codecache,exit] C1 Blobs: total=0<br>[debug][aot,codecache,exit] C2 Blobs: total=0<br>[debug][aot,codecache,exit] AOT code cache size: 894352 bytes, max entry's size: 2208 bytes<br>[info ][aot,codecache,exit] Wrote 725 AOT code entries to AOT Code Cache</div><div>Classes in AOT Cache: 12,603<br> -> KlassTrainingData: 7,101 (56.34%)<br>Objects in AOT Cache: 149,684<br> -> AOT-inited: 1,261 (0.84%)<br> -> java.lang.Class instances: 12,361 (8.26%)<br> -> java.lang.String instances: 46,320 (30.95%)<br>Methods in AOT Cache: 158,664<br> -> MethodCounters: 38,424 (24.22%)<br> -> MethodData: 33,347 (21.02%)<br> -> MethodTrainingData: 37,619 (23.71%)<br> -> CompileTrainingData: <br> -> Level 1: 552 (0.35%)<br> -> Level 2: 36 (0.02%)<br> -> Level 3: 24,737 (15.59%)<br> -> Level 4: 23,761 (14.98%)<br><br></div><div><br></div><div>Without XComp:</div><div>[debug][aot,codecache,exit] Adapters: total=724<br>[debug][aot,codecache,exit] Shared Blobs: total=0<br>[debug][aot,codecache,exit] C1 Blobs: total=0<br>[debug][aot,codecache,exit] C2 Blobs: total=0<br>[debug][aot,codecache,exit] AOT code cache size: 893136 bytes, max entry's size: 2208 bytes<br>[info ][aot,codecache,exit] Wrote 724 AOT code entries to AOT Code Cache</div><div>Classes in AOT Cache: 12,465<br> -> KlassTrainingData: 2,693 (21.60%)<br>Objects in AOT Cache: 149,416<br> -> AOT-inited: 1,250 (0.84%)<br> -> java.lang.Class instances: 12,208 (8.17%)<br> -> java.lang.String instances: 46,458 (31.09%)<br>Methods in AOT Cache: 157,933<br> -> MethodCounters: 11,004 (6.97%)<br> -> MethodData: 7,311 (4.63%)<br> -> MethodTrainingData: 8,794 (5.57%)<br> -> CompileTrainingData: <br> -> Level 1: 1,249 (0.79%)<br> -> Level 2: 947 (0.60%)<br> -> Level 3: 4,784 (3.03%)<br> -> Level 4: 1,154 (0.73%)</div><div><br></div><div>(for some reason, the log didn't say anything about any nmethod in the codecache)</div><div><br></div><div>Whatever that argument is doing, is not helping as I expected.</div><div><br></div><div>We get many more TrainingData objects, and the CompileTrainingData is done at a higher level. But it doesn't seem to speed up the application, probably because we are busy loading things we are not really going to use?</div><div><br></div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"></div></div><div>So, the conclusion is: don't bother. This looks like a dead end. María, you should have trusted the process: the JVM knows better than you.</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jan 7, 2026 at 9:18 AM María Arias de Reyna Dominguez <<a href="mailto:mariasde@redhat.com" target="_blank">mariasde@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi!</div><div><br></div><div>Thanks! I will try to take a closer look and see what is exactly what is happening.</div><div><br></div><div>Right now, a comparison on a Quarkus native vs Quarkus Leyden (J26 main latest) is close to six or seven times faster on the tests I have done. But that may be test-dependent, so I have to dig further.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jan 4, 2026 at 6:23 PM Dan Heidinga <<a href="mailto:dan.heidinga@oracle.com" target="_blank">dan.heidinga@oracle.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Happy new year!</div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif;color:rgb(0,0,0)">
<span style="font-size:12pt">> </span><span style="font-size:16px;background-color:rgb(255,255,255)">For example: a REST API. It has some initialization, port opening, reading configurations,</span></div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif;font-size:16px;color:rgb(0,0,0)">
<span style="background-color:rgb(255,255,255)">> etc... that run only once. So the code will never be trained. But it always runs at startup,</span></div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif;font-size:16px;color:rgb(0,0,0)">
<span style="background-color:rgb(255,255,255)">> impacting the time to first response.</span></div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif;font-size:16px;color:rgb(0,0,0)">
<span style="background-color:rgb(255,255,255)"><br>
</span></div>
<div dir="ltr"><span style="background-color:rgb(255,255,255)">Historically, JVMs have looked at run-once code - like the body of <clinit> - as not being worth compiling as the return on the investment in compile time is too low. There have always been
exceptions but even template style jits have avoided run once code.</span></div>
<div dir="ltr"><span style="background-color:rgb(255,255,255)"><br>
</span></div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="background-color:rgb(255,255,255)">Can you quantify how much of the applications startup is spent in these run-once methods?</span></div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif;font-size:16px;color:rgb(0,0,0)">
<span style="background-color:rgb(255,255,255)"><br>
</span></div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif;font-size:16px;color:rgb(0,0,0)">
<span style="background-color:rgb(255,255,255)">> So, how can I tell Leyden to please compile and cache those functions, even if they are</span></div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif;font-size:16px;color:rgb(0,0,0)">
<span style="background-color:rgb(255,255,255)">> going to be run just once, even if they are not optimized at all, even if those compilations</span></div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif;font-size:16px;color:rgb(0,0,0)">
<span style="background-color:rgb(255,255,255)">> can get discarded after a couple of seconds?</span></div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif;font-size:16px;color:rgb(0,0,0)">
<span style="background-color:rgb(255,255,255)"><br>
</span></div>
<div dir="ltr"><span style="background-color:rgb(255,255,255)">Compiling the code isn’t enough. There’s a lot of work with careful timing required to get the code ready for use before the first invocation. If we miss that window, then the compiled code
is just overhead.</span></div>
<div dir="ltr"><span style="background-color:rgb(255,255,255)"><br>
</span></div>
<div dir="ltr"><span style="font-size:16px;background-color:rgb(255,255,255)">For
</span><span style="background-color:rgb(255,255,255)">“</span><span style="font-size:16px;background-color:rgb(255,255,255)">expensive” or long running single use code, we may be able to precompile with C1 and get out of the interpreter earlier at
the cost of some coordination overhead to ensure the methods are installed immediately.</span></div>
<div dir="ltr" style="font-size:16px"><span style="background-color:rgb(255,255,255)"><br>
</span></div>
<div dir="ltr"><span style="background-color:rgb(255,255,255)">I think we’d need to understand better where the time is being spent to see why this run once code is slowing down startup.</span></div>
<div dir="ltr"><span style="background-color:rgb(255,255,255)"><br>
</span></div>
<div dir="ltr"><span style="background-color:rgb(255,255,255)">—</span><span style="font-size:16px;background-color:rgb(255,255,255)">Dan</span></div>
<div dir="ltr" style="font-family:Aptos,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="m_674692735790629041m_5740020876325400463m_-1912845208343392007mail-editor-reference-message-container">
<div dir="ltr"></div>
<div style="text-align:left;padding:3pt 0in 0in;border-width:1pt medium medium;border-style:solid none none;border-color:rgb(181,196,223) currentcolor currentcolor;font-family:Aptos;font-size:12pt;color:black">
<b>From: </b>leyden-dev <<a href="mailto:leyden-dev-retn@openjdk.org" target="_blank">leyden-dev-retn@openjdk.org</a>> on behalf of María Arias de Reyna Dominguez <<a href="mailto:mariasde@redhat.com" target="_blank">mariasde@redhat.com</a>><br>
<b>Date: </b>Tuesday, December 30, 2025 at 4:13 AM<br>
<b>To: </b>leyden-dev <<a href="mailto:leyden-dev@openjdk.org" target="_blank">leyden-dev@openjdk.org</a>><br>
<b>Subject: </b>Initialization code that never got trained<br>
<br>
</div>
<div dir="ltr">Happy New Year!</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">I have been doing some experiments with Leyden and realized something: there is some code at startup/initialization that never gets optimized but is impacting on startup and warmup time.</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">This was a realization while doing comparisons with native/graalvm images of the same code.</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">For example: a REST API. It has some initialization, port opening, reading configurations, etc... that run only once. So the code will never be trained. But it always runs at startup, impacting
the time to first response.</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Compared to a native image, the native image may not have it optimized, but at least it is already compiled, not interpreted. Therefore, the native image starts faster.</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">So, how can I tell Leyden to please compile and cache those functions, even if they are going to be run just once, even if they are not optimized at all, even if those compilations can
get discarded after a couple of seconds?</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Or are we just going to assume that that code, which is impacting startup time, doesn't need to be pre-compiled because we are focusing only on optimizations made by the JVM on runtime?</div>
<div dir="ltr"><br>
</div>
<div dir="ltr" class="gmail_signature">Kind regards,<br>
María Arias de Reyna Domínguez<br>
Senior Software Engineer<br>
She / Her / Hers<br>
<a href="mailto:ariasdereyna@redhat.com" target="_blank">ariasdereyna@redhat.com</a></div>
</div>
</div>
</blockquote></div>
</blockquote></div>