RFR: 8006164 [TESTBUG] compact profile hotspot test issues

[David Holmes]

On 27/08/2013 1:14 PM, Vladimir Kozlov wrote:
> On 8/26/13 7:13 PM, David Holmes wrote:
>> Hi Vladimir,
>>
>> On 27/08/2013 3:14 AM, Vladimir Kozlov wrote:
>>> Thank you, David
>>>
>>> For me it looks odd when you specify group as not other flags, like:
>>>
>>> -group:compact1
>>>
>>> Why it was designed this way?
>>
>> Are you referring to the jtreg usage? ie
>>
>> jtreg -jdk:xxx -v -r reportdir :compact1
>
> Yes, I asked about that. From the examples in TEST.groups it was not
> clear how to use them.
>
>>
>> The group is not an option it is a test designator similar to doing:
>
> It is not destination from my understanding - it is named list of tests.

Right, a group defines an explicit named list of tests, while gc/ 
"names" a list of tests implicitly by their location in the directory 
structure. Both form the "here are the tests to run" arguments to jtreg.

> So I don't see why we can't use it as flag. I understand that I am late
> and the feature already implemented in jtreg and it is only a matter how
> jtreg parses command line. But it is unusual to have command parameter
> starting with :.

jtreg needs to be able to distinguish between the name of a test and the 
name of a group hence : is a group name delimiter. Actually the complete 
syntax allows another designator in front of the :

testdir:group

which I believe supports the fact that any testdir can contain a 
TEST.ROOT file which among other things can define the group file in 
which the group definition can be found.

But this is all jtreg stuff :)

>>
>> jtreg -jdk:xxx -v -r reportdir runtime/NMT gc/
>
> Can you run subsets of tests from a group?:
>
>   jtreg -jdk:xxx -v -r reportdir :needs_jdk gc

Not at present. A group defines a set of tests, so the above would be a 
union of the tests in group :needs_jdk and the tests under directory gc.

I am currently arguing for a way to exclude a group on the command line 
so that you can "subset" to some degree eg:

jtreg gc -:needs_jdk

would run all gc tests except those listed in the needs_jdk group. This 
avoids the need to define composite groups for all permutations of tests 
that you might want to run.

BTW the way jtreg produces the set of tests in a group is to first do 
the union of all included tests and groups, then do a union of the 
excluded tests and groups, and then subtract the second set from the 
first. And each group is evaluated in isolation before being used in 
another group (as opposed to logically being inlined and then the result 
evaluated).

>>
>>> Can ou add jprt group too (as template for now)? We want for long time
>>> to run in JPRT small subset of our jtreg tests which will take less
>>> 15min. I know, it is separate problem but can you add it as template
>>> which we can use and extend later?
>>
>> Adding this is trivial. The "template" would be:
>>
>> jprt =
>>
>> There doesn't seem much point in me adding that at this time though.
>> Better to leave it until we know how it needs to be
>> defined.
>
> Okay.

Thanks. I assume you don't mind being the Reviewer for this?

It would be nice to get a second reviewer too. :)

David

> Thanks,
> Vladimir
>
>>
>> Thanks,
>> David
>>
>>> Thanks,
>>> Vladimir
>>>
>>> On 8/25/13 6:45 PM, David Holmes wrote:
>>>> This change introduces the TEST.groups file to allow jtreg to run
>>>> regression tests by groups - where the groups are defined to support
>>>> testing of compact profiles and the minimal VM.
>>>>
>>>> webrev: http://cr.openjdk.java.net/~dholmes/8006164/webrev/
>>>>
>>>> The primary groups are:
>>>>   - jdk
>>>>   - jre
>>>>   - compact3
>>>>   - compact2
>>>>   - compact2_minimal
>>>>   - compact1
>>>>   - compact1_minimal
>>>>
>>>> The minimal VM is only supported on compact1 and compact2.
>>>>
>>>> To select a group of tests you use :<group-name>
>>>>
>>>> Eg to run only those tests that can run on compact1 use:
>>>>
>>>> jtreg :compact1
>>>>
>>>> Of course you still need to point jtreg at the right kind of runtime
>>>> image (and give it a full JDK as the compile-jdk!); and if testing the
>>>> minimal VM you need to tell jtreg to select it using
>>>> -javaoptions:-minimal
>>>>
>>>> The full jtreg group facility is only available in the most recent
>>>> jtreg
>>>> builds, so you will need to grab the latest nightly build, or latest
>>>> sources.
>>>>
>>>> It is expected that these group definitions will need some tweaking. So
>>>> far testing has been limited to linux, but we want to get this in ASAP
>>>> to extend the testing.
>>>>
>>>> Note: once this is in place, anyone writing regression tests will need
>>>> to be aware of whether that test is limited to certain profiles and
>>>> update the group file accordingly. Sometimes it is not the item being
>>>> tested that determines the minimum needed profile, but the test
>>>> infrastructure eg if it uses XML.
>>>>
>>>> The changeset will be pushed via hotspot-embas that is what is used for
>>>> profile testing.
>>>>
>>>> Thanks,
>>>> David