RFR: 8257147: [TESTBUG] Set a larger default loop count for the VectorAPI jtreg tests

[Paul Sandoz]

On Thu, 10 Dec 2020 10:18:50 GMT, Xiaohong Gong <xgong at openjdk.org> wrote:

>> It's good to revisit this, but I am concerned about the increase in test execution time. I wonder if we can approach this a little differently.
>> 
>> First, I think we should switch off tiered compilation, `-XX:-TieredCompilation` (maybe reconsider later some limited form of execution with tiered compilation enabled).
>> For such a configuration I observe, when using the current invocation count and looking at inline traces, far more lines with intrinsic `VectorSupport` methods.
>> 
>> Second, we could lower the compile threshold from 10000 e.g. `-XX:CompileThreshold=1000`. We can grep the inline trace and count the lines containing intrinsic `VectorSupport` methods.
>> 
>> Using both these approaches i think we can increase intrinsification without such a large increase in test execution time.
>> 
>> WDYT?
>
> Hi @PaulSandoz , I tested the jtreg tests with these two options these days, and the results do not show better as expected. 
> 
> With `-XX:-TieredCompilation`, the tests can work well without increasing the loop count. However, I think it's not an optimal solution. First, with this flag, some issues that might only happen with tiered compilation might not be reported as expected (Currently we do have met a similar issue).  Second, a better way to add the option for the test is adding it to the test file (e.g. ` @run testng/othervm -XX:-TieredCompilation`).  However, this might override the same options people specified when running jtreg.
> 
> For `-XX:CompileThreshold=1000`, unfortunately it seems this flag doesn't have any effect that I still need to increase the loop count to make the tests effective.  Good news is that I found the flags `-XX:CompileThresholdScaling=0.1 -Xbatch` can work well even if I do not increase the loop count. However, this still increase the whole execution time a lot when running with `-conc:1`. Here is part of the comparison for the time:
> 
> Running with "-conc:10":
>              Before     After
> System A     5m21s      5m29s
> System B     5m16s      6m7s
> 
> Running with "-conc:1":
>              Before     After
> System A     33m20s     62m14s
> System B     36m17s     71m59s
> 
> So do you have tried these before? Or any better idea about this? Thanks!
> 
> Xiaohong Gong

Thanks for looking at this in more detail. Before suggesting i did try some rough experiments which did indicate some upside to running without tiered with reduced compiler thresholds, but i only selected a few test classes to run against.

It's true that some issues have been found in tiered compilation. However, i think there is a tradeoff, as these tests are really designed exercise C2. 

My suggestion for now is to just turn off tiered compilation, which gets us reasonably far, and perhaps keep pushing on compiler threshold investigation as a follow up.

 I will follow up with our Hotspot test engineers to see if there is a way to run these tests less frequently under tiered compilation (as we do for other flags to stress the HotSpot compilers, such as `-Xcomp`).

-------------

PR: https://git.openjdk.java.net/jdk/pull/1621