RFR(XS): 8200187: Exclude 3 long-running tests from tier1

[Leonid Mesnik]

Chris, David

Thank you for review. 

> On Mar 26, 2018, at 5:48 PM, Chris Plummer <chris.plummer at oracle.com> wrote:
> 
> Hi Leonid,
> 
> The exclusion of these 3 tests from tier1 looks ok to me. It did lead me to some other questions however. The first was that if you exclude them from tier1, how to they get included in a later tier. The answer, for the SA tests, is that all of serviceability is included in tier2. So the next question was whether or not we want to repeat testing in higher tiers. I suppose since higher tiers include new platforms and build options, the answer is yes, but I hope the upper tiers are constructed well enough that we aren't wasting too much time repeating the same test(s) on the same platforms with the same options.
> 

It depends from component . Runtime the groups organized by excluding previous tiers like:

hotspot_tier2_runtime = \
  runtime/ \
  serviceability/ \
 -runtime/CommandLine/OptionsValidation/TestOptionsWithRanges.java \
 -runtime/Thread/TestThreadDumpMonitorContention.java \
 -runtime/containers/ \
 -:tier1_runtime \
 -:tier1_serviceability \
 -:hotspot_tier2_runtime_platform_agnostic
…

hotspot_tier3_runtime = \
  runtime/ \
  serviceability/ \
 -runtime/containers/ \
 -:tier1_runtime \
 -:tier1_serviceability \
 -:hotspot_tier2_runtime_platform_agnostic \
 -:hotspot_tier2_runtime

… 

So once test is excluded from tier1 it automatically is executed as a part of tier 2/3. While these tiers doesn’t execute tests from tier1. The other teams organize their testing slightly different. 
However the whole idea is that test excluded from tier1 should be executed in latest tiers and test should not re-executed in fast tiers once it was done already. However during testing like PIT 
we might just run whole hotspot_[component] team even some subset was run always in CI.


Leonid

> thanks,
> 
> Chris
> 
> On 3/26/18 4:32 PM, Leonid Mesnik wrote:
>> Hi
>> 
>> There is no strict rules and time budgets for hotspot tiers. It is assumed that faster tests should be executed earlier tiers. The order and total impact of of these test depends on how they are executed.
>> 
>> Here is time of execution of tier1 groups with and without these excluded tests:
>> The whole :tier1 time reduced from 40 to 30 minutes (on dedicated HW with 32 core)
>> Time for :tier1_serviceability reduced from 15 to 6 min (in HS CI)
>> Time for :tier1_gc_1 reduced from 12 to 8-10 min  (in HS CI)
>> 
>> The benefits of exclusion of gc/g1/ihop/TestIHOPErgo.java are not so significant. However it is the only one GC stress test executed in tier1 while intention is to don’t run stress testing in tier1.
>> 
>> Tests
>> serviceability/sa/ClhsdbScanOops.java
>> serviceability/sa/TestHeapDumpForLargeArray.java
>> are now hotspot_tier3_runtime which includes all hotspot_serviceability tests which are not a part of tier1. They are executed as a part of tier3 now as well as most of serviceability tests.
>> 
>> Test gc/g1/ihop/TestIHOPErgo.java in now in group :hotspot_gc only and also it is marked as “stress” test. So it is executed with all other GC stress tests
>> 
>> Leonid
>> 
>>> On Mar 25, 2018, at 3:48 PM, David Holmes <david.holmes at oracle.com> wrote:
>>> 
>>> Hi Leonid,
>>> 
>>> On 24/03/2018 9:31 AM, Leonid Mesnik wrote:
>>>> Hi
>>>> Could you please review following fix which exclude following tests from tier1 testing:
>>>> serviceability/sa/ClhsdbScanOops.java
>>>> serviceability/sa/TestHeapDumpForLargeArray.java
>>>> gc/g1/ihop/TestIHOPErgo.java
>>>> Each of them takes more then 5 minutes to complete and significantly increase overall time to complete tier1.
>>> I'd need to see a much more detailed analysis of all the tests run, the order they run and the execution times to ascertain what impact this actually has on overall test execution time.
>>> 
>>> But assuming 5 minutes is too long for tier1, this seems okay. But the tests must still be run in some other tier, and I'm not clear where that would be now? I would expect them to move to tier 3 perhaps, depending on what the time criteria for tier 3 is.
>>> 
>>> Thanks,
>>> David
>>> 
>>>> Please let me know if there are any reasons to run these tests in tier1 despite on their execution time.
>>>> webrev:http://cr.openjdk.java.net/~lmesnik/8200187/webrev.00/
>>>> bug: https://bugs.openjdk.java.net/browse/JDK-8200187
>>>> Leonid
> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/serviceability-dev/attachments/20180327/35eddac5/attachment-0001.html>