RFR(XS): 8200187: Exclude 3 long-running tests from tier1

[Chris Plummer]

Hi Leonid,

The exclusion of these 3 tests from tier1 looks ok to me. It did lead me 
to some other questions however. The first was that if you exclude them 
from tier1, how to they get included in a later tier. The answer, for 
the SA tests, is that all of serviceability is included in tier2. So the 
next question was whether or not we want to repeat testing in higher 
tiers. I suppose since higher tiers include new platforms and build 
options, the answer is yes, but I hope the upper tiers are constructed 
well enough that we aren't wasting too much time repeating the same 
test(s) on the same platforms with the same options.

thanks,

Chris

On 3/26/18 4:32 PM, Leonid Mesnik wrote:
> Hi
>
> There is no strict rules and time budgets for hotspot tiers. It is assumed that faster tests should be executed earlier tiers. The order and total impact of of these test depends on how they are executed.
>
> Here is time of execution of tier1 groups with and without these excluded tests:
> The whole :tier1 time reduced from 40 to 30 minutes (on dedicated HW with 32 core)
> Time for :tier1_serviceability reduced from 15 to 6 min (in HS CI)
> Time for :tier1_gc_1 reduced from 12 to 8-10 min  (in HS CI)
>
> The benefits of exclusion of gc/g1/ihop/TestIHOPErgo.java are not so significant. However it is the only one GC stress test executed in tier1 while intention is to don’t run stress testing in tier1.
>
> Tests
> serviceability/sa/ClhsdbScanOops.java
> serviceability/sa/TestHeapDumpForLargeArray.java
> are now hotspot_tier3_runtime which includes all hotspot_serviceability tests which are not a part of tier1. They are executed as a part of tier3 now as well as most of serviceability tests.
>
> Test gc/g1/ihop/TestIHOPErgo.java in now in group :hotspot_gc only and also it is marked as “stress” test. So it is executed with all other GC stress tests
>
> Leonid
>
>> On Mar 25, 2018, at 3:48 PM, David Holmes <david.holmes at oracle.com> wrote:
>>
>> Hi Leonid,
>>
>> On 24/03/2018 9:31 AM, Leonid Mesnik wrote:
>>> Hi
>>> Could you please review following fix which exclude following tests from tier1 testing:
>>> serviceability/sa/ClhsdbScanOops.java
>>> serviceability/sa/TestHeapDumpForLargeArray.java
>>> gc/g1/ihop/TestIHOPErgo.java
>>> Each of them takes more then 5 minutes to complete and significantly increase overall time to complete tier1.
>> I'd need to see a much more detailed analysis of all the tests run, the order they run and the execution times to ascertain what impact this actually has on overall test execution time.
>>
>> But assuming 5 minutes is too long for tier1, this seems okay. But the tests must still be run in some other tier, and I'm not clear where that would be now? I would expect them to move to tier 3 perhaps, depending on what the time criteria for tier 3 is.
>>
>> Thanks,
>> David
>>
>>> Please let me know if there are any reasons to run these tests in tier1 despite on their execution time.
>>> webrev:http://cr.openjdk.java.net/~lmesnik/8200187/webrev.00/
>>> bug: https://bugs.openjdk.java.net/browse/JDK-8200187
>>> Leonid