RFR: 8006164 [TESTBUG] compact profile hotspot test issues

[David Holmes]

On 28/08/2013 11:49 AM, Daniel D. Daugherty wrote:
>  > http://cr.openjdk.java.net/~dholmes/8006164/webrev/
>
> test/TEST.ROOT
>      No comments.
>
> test/TEST.groups
>      Just to be clear: The addition of any new test will require this file
>      to updated? If so, then we'll have to be careful during reviews. What
>      happens if a test isn't listed? Is it by default in the 'jdk' group?

A new test only has to be explicitly listed if it requires a specific 
profile, or full JRE or full JDK to run - in which case it must be added 
to the appropriate needs_xxx group.

All tests are initially in the compact1_minimal group (and hence all the 
'higher' groups) unless explicitly excluded by being a member of a 
needs_xxx group.

Effective use of this requires some discipline but regardless of the 
mechanism (groups or tags) when you add a regression test you must now 
ask yourself "does this test have specific API or VM service 
dependencies?" - and then answer the question :) One way to answer the 
API part is to compile the test using the javac -profile option. Eg if:

javac -profile compact1 MyTest.java

compiles, then it doesn't require a specific profile/jre to run on. The 
minimal VM issue is harder to determine without actually running on the 
minimal VM (which by default we don't build for SE). Tests that require 
the full JDK tend to do so because they use tools (jcmd, jps, etc) that 
are only in the full JDK - though if such tools can be accessed from the 
compile-jdk instead then that dependency is removed.

> Thumbs up.

Thanks!

David


> Dan
>
>
> On 8/27/13 2:55 AM, Vladimir Kozlov wrote:
>> > Thanks. I assume you don't mind being the Reviewer for this?
>>
>> Yes, it is reviewed.
>>
>> Thanks,
>> Vladimir
>>
>> On 8/26/13 8:36 PM, David Holmes wrote:
>>> On 27/08/2013 1:14 PM, Vladimir Kozlov wrote:
>>>> On 8/26/13 7:13 PM, David Holmes wrote:
>>>>> Hi Vladimir,
>>>>>
>>>>> On 27/08/2013 3:14 AM, Vladimir Kozlov wrote:
>>>>>> Thank you, David
>>>>>>
>>>>>> For me it looks odd when you specify group as not other flags, like:
>>>>>>
>>>>>> -group:compact1
>>>>>>
>>>>>> Why it was designed this way?
>>>>>
>>>>> Are you referring to the jtreg usage? ie
>>>>>
>>>>> jtreg -jdk:xxx -v -r reportdir :compact1
>>>>
>>>> Yes, I asked about that. From the examples in TEST.groups it was not
>>>> clear how to use them.
>>>>
>>>>>
>>>>> The group is not an option it is a test designator similar to doing:
>>>>
>>>> It is not destination from my understanding - it is named list of
>>>> tests.
>>>
>>> Right, a group defines an explicit named list of tests, while gc/
>>> "names" a list of tests implicitly by their location
>>> in the directory structure. Both form the "here are the tests to run"
>>> arguments to jtreg.
>>>
>>>> So I don't see why we can't use it as flag. I understand that I am late
>>>> and the feature already implemented in jtreg and it is only a matter
>>>> how
>>>> jtreg parses command line. But it is unusual to have command parameter
>>>> starting with :.
>>>
>>> jtreg needs to be able to distinguish between the name of a test and
>>> the name of a group hence : is a group name
>>> delimiter. Actually the complete syntax allows another designator in
>>> front of the :
>>>
>>> testdir:group
>>>
>>> which I believe supports the fact that any testdir can contain a
>>> TEST.ROOT file which among other things can define the
>>> group file in which the group definition can be found.
>>>
>>> But this is all jtreg stuff :)
>>>
>>>>>
>>>>> jtreg -jdk:xxx -v -r reportdir runtime/NMT gc/
>>>>
>>>> Can you run subsets of tests from a group?:
>>>>
>>>>   jtreg -jdk:xxx -v -r reportdir :needs_jdk gc
>>>
>>> Not at present. A group defines a set of tests, so the above would be
>>> a union of the tests in group :needs_jdk and the
>>> tests under directory gc.
>>>
>>> I am currently arguing for a way to exclude a group on the command
>>> line so that you can "subset" to some degree eg:
>>>
>>> jtreg gc -:needs_jdk
>>>
>>> would run all gc tests except those listed in the needs_jdk group.
>>> This avoids the need to define composite groups for
>>> all permutations of tests that you might want to run.
>>>
>>> BTW the way jtreg produces the set of tests in a group is to first do
>>> the union of all included tests and groups, then
>>> do a union of the excluded tests and groups, and then subtract the
>>> second set from the first. And each group is
>>> evaluated in isolation before being used in another group (as opposed
>>> to logically being inlined and then the result
>>> evaluated).
>>>
>>>>>
>>>>>> Can ou add jprt group too (as template for now)? We want for long
>>>>>> time
>>>>>> to run in JPRT small subset of our jtreg tests which will take less
>>>>>> 15min. I know, it is separate problem but can you add it as template
>>>>>> which we can use and extend later?
>>>>>
>>>>> Adding this is trivial. The "template" would be:
>>>>>
>>>>> jprt =
>>>>>
>>>>> There doesn't seem much point in me adding that at this time though.
>>>>> Better to leave it until we know how it needs to be
>>>>> defined.
>>>>
>>>> Okay.
>>>
>>> Thanks. I assume you don't mind being the Reviewer for this?
>>>
>>> It would be nice to get a second reviewer too. :)
>>>
>>> David
>>>
>>>> Thanks,
>>>> Vladimir
>>>>
>>>>>
>>>>> Thanks,
>>>>> David
>>>>>
>>>>>> Thanks,
>>>>>> Vladimir
>>>>>>
>>>>>> On 8/25/13 6:45 PM, David Holmes wrote:
>>>>>>> This change introduces the TEST.groups file to allow jtreg to run
>>>>>>> regression tests by groups - where the groups are defined to support
>>>>>>> testing of compact profiles and the minimal VM.
>>>>>>>
>>>>>>> webrev: http://cr.openjdk.java.net/~dholmes/8006164/webrev/
>>>>>>>
>>>>>>> The primary groups are:
>>>>>>>   - jdk
>>>>>>>   - jre
>>>>>>>   - compact3
>>>>>>>   - compact2
>>>>>>>   - compact2_minimal
>>>>>>>   - compact1
>>>>>>>   - compact1_minimal
>>>>>>>
>>>>>>> The minimal VM is only supported on compact1 and compact2.
>>>>>>>
>>>>>>> To select a group of tests you use :<group-name>
>>>>>>>
>>>>>>> Eg to run only those tests that can run on compact1 use:
>>>>>>>
>>>>>>> jtreg :compact1
>>>>>>>
>>>>>>> Of course you still need to point jtreg at the right kind of runtime
>>>>>>> image (and give it a full JDK as the compile-jdk!); and if
>>>>>>> testing the
>>>>>>> minimal VM you need to tell jtreg to select it using
>>>>>>> -javaoptions:-minimal
>>>>>>>
>>>>>>> The full jtreg group facility is only available in the most recent
>>>>>>> jtreg
>>>>>>> builds, so you will need to grab the latest nightly build, or latest
>>>>>>> sources.
>>>>>>>
>>>>>>> It is expected that these group definitions will need some
>>>>>>> tweaking. So
>>>>>>> far testing has been limited to linux, but we want to get this in
>>>>>>> ASAP
>>>>>>> to extend the testing.
>>>>>>>
>>>>>>> Note: once this is in place, anyone writing regression tests will
>>>>>>> need
>>>>>>> to be aware of whether that test is limited to certain profiles and
>>>>>>> update the group file accordingly. Sometimes it is not the item
>>>>>>> being
>>>>>>> tested that determines the minimum needed profile, but the test
>>>>>>> infrastructure eg if it uses XML.
>>>>>>>
>>>>>>> The changeset will be pushed via hotspot-embas that is what is
>>>>>>> used for
>>>>>>> profile testing.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> David
>>
>>
>