RFR: 8231981: Refactor test definitions to split RT and SVC tests

[David Holmes]

On 13/10/2019 1:21 pm, Leonid Mesnik wrote:
>> On Oct 12, 2019, at 7:34 PM, David Holmes <david.holmes at oracle.com> wrote:
>>
>> On 13/10/2019 3:27 am, Leonid Mesnik wrote:
>>> I think you messed tier3 test group and tier3 task definitions in our internal test system. They are not related. jtreg tiers groups don't define any vm flags.
>>
>> Yes I know the jtreg groups don't define flags, but the tier task definitions do and they also map to jtreg groups. That said I was mis-reading the group definitions.
>>
>>> The hotspot_tier3_runtime excludes all hotspot_tier2_runtime, so it means that hotspot_tier3_runtime doesn't include any of serviceability tests.
>>>   404 hotspot_tier3_runtime = \
>>>   405   runtime/ \
>>>   406  -:tier1_runtime \
>>>   407  -:hotspot_tier2_runtime_platform_agnostic \
>>>   408  -:hotspot_tier2_runtime
>>
>> I was looking at the original form:
>>
>> 402 hotspot_tier3_runtime = \
>> 403   runtime/ \
>> 404   serviceability/ \
>> 405  -:tier1_runtime \
>> 406  -:tier1_serviceability \
>> 407  -:hotspot_tier2_runtime_platform_agnostic \
>> 408  -:hotspot_tier2_runtime
>>
>> which is basically saying that for serviceability we would automatically run all tests in tier3 that had not previously been run in tiers 1 or 2. Now it may be that is currently an empty set, but it would mean that if any tests were later excluded from tier2 then they would still be in tier3. With the new format there is no tier3 for serviceability, so if any tier2 test gets excluded from a group (because it is too slow for example) then it will simply be dropped, unless the person doing the exclusion realizes they need to add another group to catch it for a later tier.
>>
>> Ideally I think we would use this kind of pattern up through at least tier4. But I suspect if a resulting group is empty we may run foul of jtreg's "no tests selected to run" error. :(
> 
> You are right. The tests might be just dropped if excluded from hotspot_tier2_serviceability. And you are also correct that empty group for tier3 cause execution errors. It is exactly the same for runtime tier3. Compiler and gc are slightly different.
> 
> I believe that person who excluding tests from  hotspot_tier2_serviceability  should carefully verify that are tested anywhere and add hotspot_tier3_serviceability if needed.

Okay - lets hope that is the case.

Thanks,
David

> Leonid
>>
>> David
>>
>>> So we need to run hotspot_tier2_serviceability only same way as hotspot_tier2_runtime to get some coverage.
>>> Leonid
>>>> On Oct 12, 2019, at 5:49 AM, David Holmes <david.holmes at oracle.com <mailto:david.holmes at oracle.com>> wrote:
>>>>
>>>> On 12/10/2019 5:34 pm, Leonid Mesnik wrote:
>>>>> There is nothing to run in tier3. Tier2 includes all serviceability tests.
>>>>> hotspot_tier2_serviceability = \
>>>>>    serviceability/ \
>>>>>   -:tier1_serviceability
>>>>
>>>> Doesn't tier3 apply some different flags ?
>>>>
>>>> David
>>>>
>>>>> Leonid
>>>>>> On Oct 12, 2019, at 12:30 AM, David Holmes <david.holmes at oracle.com <mailto:david.holmes at oracle.com>> wrote:
>>>>>>
>>>>>> Hi Leonid,
>>>>>>
>>>>>> On 12/10/2019 1:28 pm, Leonid Mesnik wrote:
>>>>>>> Thank you.
>>>>>>> Could anyone from runtime look on this also?
>>>>>>
>>>>>> I'm somewhat confused. Presently the serviceability tests run as part of hotspot_tier2_runtime and hotspot_tier3_runtime, but with this change they will only run in tier2.
>>>>>>
>>>>>> David
>>>>>>
>>>>>>> Leonid
>>>>>>>> On Oct 11, 2019, at 5:40 PM, serguei.spitsyn at oracle.com <mailto:serguei.spitsyn at oracle.com> wrote:
>>>>>>>>
>>>>>>>> Hi Leonid,
>>>>>>>>
>>>>>>>> Looks good.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Serguei
>>>>>>>>
>>>>>>>> On 10/11/19 4:27 PM, Leonid Mesnik wrote:
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> Could you please review following fix which separate runtime and serviceability tests. Runtime and serviceability belong to different components and separating tier2/3 separate failures and helps to analyze results.
>>>>>>>>>
>>>>>>>>> webrev: http://cr.openjdk.java.net/~lmesnik/8231981/webrev.00/ <http://cr.openjdk.java.net/~lmesnik/8231981/webrev.00/>
>>>>>>>>> bug https://bugs.openjdk.java.net/browse/JDK-8231981 <https://bugs.openjdk.java.net/browse/JDK-8231981>
>>>>>>>>>
>>>>>>>>> Leonid
>>>>>>>>
>