RFR: 8231981: Refactor test definitions to split RT and SVC tests

[David Holmes]

On 13/10/2019 3:27 am, Leonid Mesnik wrote:
> I think you messed tier3 test group and tier3 task definitions in our 
> internal test system. They are not related. jtreg tiers groups don't 
> define any vm flags.

Yes I know the jtreg groups don't define flags, but the tier task 
definitions do and they also map to jtreg groups. That said I was 
mis-reading the group definitions.

> The hotspot_tier3_runtime excludes all hotspot_tier2_runtime, so it 
> means that hotspot_tier3_runtime doesn't include any of serviceability 
> tests.
> 
>   404 hotspot_tier3_runtime = \
>   405   runtime/ \
>   406  -:tier1_runtime \
>   407  -:hotspot_tier2_runtime_platform_agnostic \
>   408  -:hotspot_tier2_runtime

I was looking at the original form:

  402 hotspot_tier3_runtime = \
  403   runtime/ \
  404   serviceability/ \
  405  -:tier1_runtime \
  406  -:tier1_serviceability \
  407  -:hotspot_tier2_runtime_platform_agnostic \
  408  -:hotspot_tier2_runtime

which is basically saying that for serviceability we would automatically 
run all tests in tier3 that had not previously been run in tiers 1 or 2. 
Now it may be that is currently an empty set, but it would mean that if 
any tests were later excluded from tier2 then they would still be in 
tier3. With the new format there is no tier3 for serviceability, so if 
any tier2 test gets excluded from a group (because it is too slow for 
example) then it will simply be dropped, unless the person doing the 
exclusion realizes they need to add another group to catch it for a 
later tier.

Ideally I think we would use this kind of pattern up through at least 
tier4. But I suspect if a resulting group is empty we may run foul of 
jtreg's "no tests selected to run" error. :(

David

> 
> So we need to run hotspot_tier2_serviceability only same way as 
> hotspot_tier2_runtime to get some coverage.
> 
> Leonid
> 
>> On Oct 12, 2019, at 5:49 AM, David Holmes <david.holmes at oracle.com 
>> <mailto:david.holmes at oracle.com>> wrote:
>>
>> On 12/10/2019 5:34 pm, Leonid Mesnik wrote:
>>> There is nothing to run in tier3. Tier2 includes all serviceability 
>>> tests.
>>> hotspot_tier2_serviceability = \
>>>   serviceability/ \
>>>  -:tier1_serviceability
>>
>> Doesn't tier3 apply some different flags ?
>>
>> David
>>
>>> Leonid
>>>> On Oct 12, 2019, at 12:30 AM, David Holmes <david.holmes at oracle.com 
>>>> <mailto:david.holmes at oracle.com>> wrote:
>>>>
>>>> Hi Leonid,
>>>>
>>>> On 12/10/2019 1:28 pm, Leonid Mesnik wrote:
>>>>> Thank you.
>>>>> Could anyone from runtime look on this also?
>>>>
>>>> I'm somewhat confused. Presently the serviceability tests run as 
>>>> part of hotspot_tier2_runtime and hotspot_tier3_runtime, but with 
>>>> this change they will only run in tier2.
>>>>
>>>> David
>>>>
>>>>> Leonid
>>>>>> On Oct 11, 2019, at 5:40 PM, serguei.spitsyn at oracle.com 
>>>>>> <mailto:serguei.spitsyn at oracle.com> wrote:
>>>>>>
>>>>>> Hi Leonid,
>>>>>>
>>>>>> Looks good.
>>>>>>
>>>>>> Thanks,
>>>>>> Serguei
>>>>>>
>>>>>> On 10/11/19 4:27 PM, Leonid Mesnik wrote:
>>>>>>> Hi
>>>>>>>
>>>>>>> Could you please review following fix which separate runtime and 
>>>>>>> serviceability tests. Runtime and serviceability belong to 
>>>>>>> different components and separating tier2/3 separate failures and 
>>>>>>> helps to analyze results.
>>>>>>>
>>>>>>> webrev: http://cr.openjdk.java.net/~lmesnik/8231981/webrev.00/ 
>>>>>>> <http://cr.openjdk.java.net/~lmesnik/8231981/webrev.00/>
>>>>>>> bug https://bugs.openjdk.java.net/browse/JDK-8231981 
>>>>>>> <https://bugs.openjdk.java.net/browse/JDK-8231981>
>>>>>>>
>>>>>>> Leonid
>>>>>>
>