premain: performance sniff tests

Thu Sep 7 02:45:13 UTC 2023

>     There were some experiments with PetClinic on our side before and it was
>     noticed that the application relies on custom loaders which aren't fully
>     supported yet.
> 
> 
> Can you please elaborate more about the support required for handling 
> custom classloaders.
> Do they have an impact on AOT code quality or the training data?

As of now, there are a number of implementation-specific constraints 
imposed to simplify prototyping and experiments. I'm not sure about 
training data, but code caching is conservatively disabled for all 
classes loaded by custom loaders. Also, some recent CDS-specific 
enhancements in Leyden repository (like class preloading, constant pool 
entries pre-resolution) are disabled for custom loaders. So, as of 
today, code loaded by custom loaders doesn't benefit much from the work 
in Leyden.

>     Until proper support for custom loaders is there, I suggest to modify
>     the benchmark so it relies only on existing system loaders.
> 
> 
> Is there ongoing work to improve the support for custom loaders?

I'll let Ioi to comment on the plans about custom loaders. But I 
wouldn't expect much progress on that front in the short term.

> Another thing that I want to check is the portability of the AOT code.
> Do we do anything to ensure the AOT code is portable across 
> microarchitectures,
> that is, it is not tied to the CPU features of the system where the code 
> is being generated.
> If we bundle the cached code archive in containers, which I expect would 
> be one of the ways to deploy these archives,
> then the portability would come into picture.

We plan to address that in the future, but, so far, JVM does nothing 
special in that respect. Moreover, CDS archive itself imposes 
constraints on supported JVM modes (e.g., compressed oops and compressed 
class pointer modes should match). But users are free to specify any 
additional constraints during training process. For example, if shared 
code archive is generated with -XX:UseAVX=2, there won't be any AVX512 
instructions present in the archived code which makes it safe to run on 
any AVX2-capable hardware.

Best regards,
Vladimir Ivanov

> On Tue, Sep 5, 2023 at 8:41 PM Vladimir Ivanov 
> <vladimir.x.ivanov at oracle.com <mailto:vladimir.x.ivanov at oracle.com>> wrote:
> 
>     Hi Ashutosh,
> 
>     Thanks for giving it a try!
> 
>     There were some experiments with PetClinic on our side before and it
>     was
>     noticed that the application relies on custom loaders which aren't
>     fully
>     supported yet. It was the main limiting factor for new optimizations.
>     Until proper support for custom loaders is there, I suggest to modify
>     the benchmark so it relies only on existing system loaders.
> 
>     Speaking of peak performance, some loss of performance is expected.
>     Cached code is compiled conservatively (e.g., no constant folding for
>     static final fields) so it can be reused in deployment runs. For now,
>     the intended solution is to eventually recompile cached code online
>     with
>     all the optimizations enabled (have to be explicitly enabled
>     -XX:+UseRecompilation). It's a work-in-progress and our experience
>     using
>     it was mixed: recompilation doesn't always fully restore peak
>     performance.
> 
>     But assuming that both CDS and cached code archive are underutilized
>     (due to aforementioned reliance on custom loaders), 10% sounds way too
>     big of a difference. I suggest to experiment with different flag
>     combinations (e.g., turning ReplayTraining and LoadCachedCode on and
>     off
>     independently).
> 
>     There's additional diagnostic output JVM produces which may help to
>     observe effects from new optimizations during both training and
>     deployment runs:
> 
>        * -XX:+PrintCompilation: compilations satisfied from cached code
>     archive are marked w/ "R";
> 
>        * -XX:+CITime:  prints information about cached code archive usage;
> 
>        * -Xlog:init=info: produces additional information about some
>     startup
>     activities
> 
>        * -XX:+PrintSharedArchiveAndExit additionally dumps training data
>     and
>     cached code archive info
> 
>        * -Xlog:scc*=info and -Xlog:cds*=info print lots of additional
>     information both during training and deployment
> 
>     Hope it helps.
> 
>     Best regards,
>     Vladimir Ivanov
> 
>     On 9/5/23 13:52, Ashutosh Mehra wrote:
>      > Hi,
>      >
>      > We have been interested in persisting the profiling data in the CDS
>      > archive with the intention of improving the application's warmup
>     time.
>      > And now that the premain branch is here that does save profile data
>      > along with AOT, we started playing with the premain branch to
>     understand
>      > its impact on the performance.
>      >
>      > Our setup uses Springboot Petclinic [0] application and the CDS and
>      > shared code archives are generated in a manner similar to this
>     script [1].
>      > Our training run only covers the application startup phase. That
>     means
>      > at each step we start the application and shut it down without
>     putting
>      > any load on it.
>      >
>      > Using the archives thus generated I have done few experiments on my
>      > local system. In these experiments the application is bound to
>     two cpus.
>      > The baseline for comparing the results is the case where the CDS
>     archive
>      > does not have any profiling data and there is no shared code archive.
>      > The "premain" configuration refers to using a shared code archive
>     and a
>      > CDS archive with training data.
>      >
>      > Here are some initial results:
>      >
>      > 1. Startup: It is heartening to see start-up time improve by
>     almost 11%.
>      >
>      > baseline       10.2s
>      > premain         9.1s
>      >
>      > 2. Warmup:
>      > This test measures the warmup time by applying load using 1 jmeter
>      > thread to get an idea of the ramp-up time to reach the peak
>     throughput.
>      > The load is applied for the duration of 300 seconds. The graph
>     [2] for
>      > aot+profiling configuration shows interesting behavior.
>      > In the initial period premain is ramping up faster than the
>     baseline.
>      > Then the slope of the curve for premain reduces significantly and a
>      > couple of dips are also seen. Finally the throughput stabilizes.
>      > It shows a drastic difference in the warmup time of the
>     application when
>      > running with the "premain" config.
>      >
>      > 3. Peak throughput: Last experiment is to measure peak
>     throughput. It
>      > starts with a warm-up phase of 180 seconds using 1 jmeter thread.
>     After
>      > the warmup phase the load is applied with 10 jmeter threads for a
>      > duration of 5 mins.
>      > Last two minutes of throughput is considered for measurement. The
>     graph
>      > [3] for this test shows almost a 10% drop in the throughput
>     compared to
>      > the baseline.
>      >
>      >
>      > I am sure others would have done similar testing.  My questions are:
>      >
>      > 1. Are these results on the expected lines?
>      > 2. Are these tests using the CDS and the shared code (or cached
>     code)
>      > archives in the expected manner.
>      > 3. Warmup time with the premain branch looks pretty bad which is
>      > surprising. Is there any trick I missed in my tests? Is there
>     anything
>      > else that needs to be done to get better warmup time?
>      > 4. What is the point of creating a new static archive? Shouldn't the
>      > applications just create the dynamic archive?
>      > 5. I am also wondering if there is any design doc that can be shared
>      > that explains the AOT compilation strategy adopted in the premain
>     branch?
>      >
>      > I have placed my scripts here [4] in case anyone wants to use
>     them to
>      > run these tests (you need to build the Petclinic app before using
>     these
>      > scripts).
>      >
>      > Please feel free to share your thoughts.
>      >
>      > [0] https://github.com/spring-projects/spring-petclinic
>     <https://urldefense.com/v3/__https://github.com/spring-projects/spring-petclinic__;!!ACWV5N9M2RV99hQ!PpMTtwDD2_k-drLo0lLtZ1pybI_zZkMM7RH-TfRvfAEBwceCBkjYqfi7baTqI_r0e5f-qUZenTYkntGa1ZwDrL4$>
>      > <https://github.com/spring-projects/spring-petclinic
>     <https://urldefense.com/v3/__https://github.com/spring-projects/spring-petclinic__;!!ACWV5N9M2RV99hQ!PpMTtwDD2_k-drLo0lLtZ1pybI_zZkMM7RH-TfRvfAEBwceCBkjYqfi7baTqI_r0e5f-qUZenTYkntGa1ZwDrL4$>>
>      > [1]
>      >
>     https://github.com/openjdk/leyden/blob/d960fb15258cc99a1bf7f0b1e94bd8be06605aad/test/hotspot/jtreg/premain/lib/premain-run.sh#L70-L101 <https://urldefense.com/v3/__https://github.com/openjdk/leyden/blob/d960fb15258cc99a1bf7f0b1e94bd8be06605aad/test/hotspot/jtreg/premain/lib/premain-run.sh*L70-L101__;Iw!!ACWV5N9M2RV99hQ!PpMTtwDD2_k-drLo0lLtZ1pybI_zZkMM7RH-TfRvfAEBwceCBkjYqfi7baTqI_r0e5f-qUZenTYkntGa4rgAjhQ$> <https://github.com/openjdk/leyden/blob/d960fb15258cc99a1bf7f0b1e94bd8be06605aad/test/hotspot/jtreg/premain/lib/premain-run.sh#L70-L101 <https://urldefense.com/v3/__https://github.com/openjdk/leyden/blob/d960fb15258cc99a1bf7f0b1e94bd8be06605aad/test/hotspot/jtreg/premain/lib/premain-run.sh*L70-L101__;Iw!!ACWV5N9M2RV99hQ!PpMTtwDD2_k-drLo0lLtZ1pybI_zZkMM7RH-TfRvfAEBwceCBkjYqfi7baTqI_r0e5f-qUZenTYkntGa4rgAjhQ$>>
>      > [2]
>      >
>     https://github.com/ashu-mehra/leyden-perf/blob/main/spring/fd82682/tput-t1.svg <https://urldefense.com/v3/__https://github.com/ashu-mehra/leyden-perf/blob/main/spring/fd82682/tput-t1.svg__;!!ACWV5N9M2RV99hQ!PpMTtwDD2_k-drLo0lLtZ1pybI_zZkMM7RH-TfRvfAEBwceCBkjYqfi7baTqI_r0e5f-qUZenTYkntGaB0ccgkk$> <https://github.com/ashu-mehra/leyden-perf/blob/main/spring/fd82682/tput-t1.svg <https://urldefense.com/v3/__https://github.com/ashu-mehra/leyden-perf/blob/main/spring/fd82682/tput-t1.svg__;!!ACWV5N9M2RV99hQ!PpMTtwDD2_k-drLo0lLtZ1pybI_zZkMM7RH-TfRvfAEBwceCBkjYqfi7baTqI_r0e5f-qUZenTYkntGaB0ccgkk$>>
>      > [3]
>      >
>     https://github.com/ashu-mehra/leyden-perf/blob/main/spring/fd82682/tput-t10.svg <https://urldefense.com/v3/__https://github.com/ashu-mehra/leyden-perf/blob/main/spring/fd82682/tput-t10.svg__;!!ACWV5N9M2RV99hQ!PpMTtwDD2_k-drLo0lLtZ1pybI_zZkMM7RH-TfRvfAEBwceCBkjYqfi7baTqI_r0e5f-qUZenTYkntGagpmiT9g$> <https://github.com/ashu-mehra/leyden-perf/blob/main/spring/fd82682/tput-t10.svg <https://urldefense.com/v3/__https://github.com/ashu-mehra/leyden-perf/blob/main/spring/fd82682/tput-t10.svg__;!!ACWV5N9M2RV99hQ!PpMTtwDD2_k-drLo0lLtZ1pybI_zZkMM7RH-TfRvfAEBwceCBkjYqfi7baTqI_r0e5f-qUZenTYkntGagpmiT9g$>>
>      > [4] https://github.com/ashu-mehra/leyden-perf
>     <https://urldefense.com/v3/__https://github.com/ashu-mehra/leyden-perf__;!!ACWV5N9M2RV99hQ!PpMTtwDD2_k-drLo0lLtZ1pybI_zZkMM7RH-TfRvfAEBwceCBkjYqfi7baTqI_r0e5f-qUZenTYkntGaY_ZcbT4$>
>      > <https://github.com/ashu-mehra/leyden-perf
>     <https://urldefense.com/v3/__https://github.com/ashu-mehra/leyden-perf__;!!ACWV5N9M2RV99hQ!PpMTtwDD2_k-drLo0lLtZ1pybI_zZkMM7RH-TfRvfAEBwceCBkjYqfi7baTqI_r0e5f-qUZenTYkntGaY_ZcbT4$>>
>      >
>      > Thanks,
>      > - Ashutosh Mehra
> 