Does it make sense to tweak compilation for training?
Hi! While searching for good documentation to link to when explaining Leyden, I found this (rather old) page: https://www.ibm.com/docs/en/sdk-java-technology/8?topic=options-xjit-xnojit with compilation options I didn't know existed. And I was wondering: does it make sense to force some things during training run to make sure we get the best training? I'm thinking for example on forcing a high level of compilation on some methods, so we arrive to production with more things optimized. Or excluding some "testing framework" methods from compilation. Or is it better not to touch anything and let Java run normally because this may become too unpredictable? Or... should I play with this and see what happens because we don't really know? :) Kind regards, María Arias de Reyna Domínguez Senior Software Engineer She / Her / Hers ariasdereyna@redhat.com
That’s a great question - restating my understanding of it: “Is it useful to hardcode specific JIT options that compile early, force methods to be compiled, etc while training?” We’ve always said that training should match production as closely as possible. The more closely they match (ie: training in production using canary deployments) the better as the training produces training data that exactly matches what the JVM is doing during startup and warmup (and will likely continue to do in future deployments). If we tweak the JIT settings from the side for training, we may distort the training run in ways that are less useful and result in us missing training data (like profiles) that would be useful if we need to deopt+recompile in the production run. That’s my long way of saying if you wouldn’t deploy those options on your production runs, you probably don’t want them on your training runs either. —Dan From: leyden-dev <leyden-dev-retn@openjdk.org> on behalf of María Arias de Reyna Dominguez <mariasde@redhat.com> Date: Monday, December 15, 2025 at 4:36 AM To: leyden-dev <leyden-dev@openjdk.org> Subject: Does it make sense to tweak compilation for training? Hi! While searching for good documentation to link to when explaining Leyden, I found this (rather old) page: https://www.ibm.com/docs/en/sdk-java-technology/8?topic=options-xjit-xnojit with compilation options I didn't know existed. And I was wondering: does it make sense to force some things during training run to make sure we get the best training? I'm thinking for example on forcing a high level of compilation on some methods, so we arrive to production with more things optimized. Or excluding some "testing framework" methods from compilation. Or is it better not to touch anything and let Java run normally because this may become too unpredictable? Or... should I play with this and see what happens because we don't really know? :) Kind regards, María Arias de Reyna Domínguez Senior Software Engineer She / Her / Hers ariasdereyna@redhat.com<mailto:ariasdereyna@redhat.com>
I agree with Dan. We don't want users to force compilation of methods, rather we should let that happen organically. As for excluding the test framework from compilation, we already have options to do that in the premain branch. For instance there is DontPrecompile to exclude a method from AOT compilation in the assembly phase. There is IgnorePrecompiled to ignore the AOT code for a method in the production run. Apart from the canary deployments, there is another scenario where the users may have an offline training run and drive it through the synthetic load. In such a scenario, it may make sense to lower the execution count threshold that triggers compilation to reduce the execution time of the training run required to get a good quality cache. For example if the user knows beforehand a hot code path of the application, then the compilation of this code path can be triggered early by reducing the threshold. Theoretically this sounds useful, but in practice it can be difficult to do this right. @Maria, btw the link you referred to is for IBM SDK which is likely based on OpenJ9. I don't think these options exist for Hotspot. But there may be options in Hotspot with similar behavior. My understanding is most of these options are for diagnostic purposes, not for end users to tune their application. Thanks, - Ashutosh Mehra On Mon, Dec 15, 2025 at 10:58 AM Dan Heidinga <dan.heidinga@oracle.com> wrote:
That’s a great question - restating my understanding of it: “Is it useful to hardcode specific JIT options that compile early, force methods to be compiled, etc while training?”
We’ve always said that training should match production as closely as possible. The more closely they match (ie: training in production using canary deployments) the better as the training produces training data that exactly matches what the JVM is doing during startup and warmup (and will likely continue to do in future deployments).
If we tweak the JIT settings from the side for training, we may distort the training run in ways that are less useful and result in us missing training data (like profiles) that would be useful if we need to deopt+recompile in the production run.
That’s my long way of saying if you wouldn’t deploy those options on your production runs, you probably don’t want them on your training runs either.
—Dan
*From: *leyden-dev <leyden-dev-retn@openjdk.org> on behalf of María Arias de Reyna Dominguez <mariasde@redhat.com> *Date: *Monday, December 15, 2025 at 4:36 AM *To: *leyden-dev <leyden-dev@openjdk.org> *Subject: *Does it make sense to tweak compilation for training?
Hi!
While searching for good documentation to link to when explaining Leyden, I found this (rather old) page: https://www.ibm.com/docs/en/sdk-java-technology/8?topic=options-xjit-xnojit with compilation options I didn't know existed.
And I was wondering: does it make sense to force some things during training run to make sure we get the best training? I'm thinking for example on forcing a high level of compilation on some methods, so we arrive to production with more things optimized. Or excluding some "testing framework" methods from compilation.
Or is it better not to touch anything and let Java run normally because this may become too unpredictable?
Or... should I play with this and see what happens because we don't really know? :)
Kind regards, María Arias de Reyna Domínguez Senior Software Engineer She / Her / Hers ariasdereyna@redhat.com
I also agree with Dan that training run should match as much as possible to production runs regarding VM options. There are a LOT of JIT compiler options in HotSpot VM. But the only option we are planning to use is to support AOT portability which will define which set of CPU instructions could be used by JIT compilers during training and assembly phase. I don't think lowering threshold for compilation will give you better code. But may be we can consider lowering threshold when we create MDO. Currently MDO creation is targeting stable peek performance and created later. May be for AOT code we need separate MDO to collect profiling data for startup. Thanks, Vladimir K On 12/15/25 8:51 AM, Ashutosh Mehra wrote:
I agree with Dan. We don't want users to force compilation of methods, rather we should let that happen organically.
As for excluding the test framework from compilation, we already have options to do that in the premain branch. For instance there is DontPrecompile to exclude a method from AOT compilation in the assembly phase. There is IgnorePrecompiled to ignore the AOT code for a method in the production run.
Apart from the canary deployments, there is another scenario where the users may have an offline training run and drive it through the synthetic load. In such a scenario, it may make sense to lower the execution count threshold that triggers compilation to reduce the execution time of the training run required to get a good quality cache. For example if the user knows beforehand a hot code path of the application, then the compilation of this code path can be triggered early by reducing the threshold. Theoretically this sounds useful, but in practice it can be difficult to do this right.
@Maria, btw the link you referred to is for IBM SDK which is likely based on OpenJ9. I don't think these options exist for Hotspot. But there may be options in Hotspot with similar behavior. My understanding is most of these options are for diagnostic purposes, not for end users to tune their application.
Thanks, - Ashutosh Mehra
On Mon, Dec 15, 2025 at 10:58 AM Dan Heidinga <dan.heidinga@oracle.com <mailto:dan.heidinga@oracle.com>> wrote:
That’s a great question - restating my understanding of it: “Is it useful to hardcode specific JIT options that compile early, force methods to be compiled, etc while training?”
We’ve always said that training should match production as closely as possible. The more closely they match (ie: training in production using canary deployments) the better as the training produces training data that exactly matches what the JVM is doing during startup and warmup (and will likely continue to do in future deployments).
If we tweak the JIT settings from the side for training, we may distort the training run in ways that are less useful and result in us missing training data (like profiles) that would be useful if we need to deopt+recompile in the production run.
That’s my long way of saying if you wouldn’t deploy those options on your production runs, you probably don’t want them on your training runs either.
—Dan
*From: *leyden-dev <leyden-dev-retn@openjdk.org <mailto:leyden-dev- retn@openjdk.org>> on behalf of María Arias de Reyna Dominguez <mariasde@redhat.com <mailto:mariasde@redhat.com>> *Date: *Monday, December 15, 2025 at 4:36 AM *To: *leyden-dev <leyden-dev@openjdk.org <mailto:leyden- dev@openjdk.org>> *Subject: *Does it make sense to tweak compilation for training?
Hi!
While searching for good documentation to link to when explaining Leyden, I found this (rather old) page: https://www.ibm.com/docs/en/ sdk-java-technology/8?topic=options-xjit-xnojit <https:// www.ibm.com/docs/en/sdk-java-technology/8?topic=options-xjit- xnojit> with compilation options I didn't know existed.
And I was wondering: does it make sense to force some things during training run to make sure we get the best training? I'm thinking for example on forcing a high level of compilation on some methods, so we arrive to production with more things optimized. Or excluding some "testing framework" methods from compilation.
Or is it better not to touch anything and let Java run normally because this may become too unpredictable?
Or... should I play with this and see what happens because we don't really know? :)
Kind regards, María Arias de Reyna Domínguez Senior Software Engineer She / Her / Hers ariasdereyna@redhat.com <mailto:ariasdereyna@redhat.com>
participants (4)
-
Ashutosh Mehra
-
Dan Heidinga
-
María Arias de Reyna Dominguez
-
Vladimir Kozlov