JDK 8 b118 ea test results are now available

[Balchandra Vaidya]

On 11/12/2013 17:55, Jonathan Gibbons wrote:
> Balchandra,
>
> The important part of that of interest here is the part about "calls 
> jtreg commands
> (one each for jdk, langtools and hotspot) with those recommended
> options".
>
> Is there a way you could post here the segment of the script that does 
> that?
>
Here it is

2.1 Running tests in jdk/test
$ jtreg -dir:{openjdk source top directory}/jdk/test -verbose:summary 
-exclude:{openjdk source top directory}/jdk/test/ProblemList.txt 
-conc:auto -a -ignore:quiet -timeoutFactor:5 -othervm -testjdk:{location 
of the test jdk} `cat dir.list 
<http://download.java.net/jdk8/testresults/docs/dir.list>`

2.2 Running tests in langtools/test
$ jtreg -dir:{openjdk source top directory}/langtools/test 
-verbose:summary -conc:auto -a -ignore:quiet -timeoutFactor:5 -agentvm 
-testjdk:{location of the test jdk} com tools

2.3 Running tests in hotspot/test
$ jtreg -dir:{openjdk source top directory}/hotspot/test 
-verbose:summary -conc:auto -a -ignore:quiet -timeoutFactor:5 -agentvm 
-testjdk:{location of the test jdk} compiler gc runtime sanity 
serviceability


The instructions is at 
http://download.java.net/jdk8/testresults/docs/howtoruntests.html
and is linked from 
http://www.java.net/download/jdk8/testresults/testresults.html

The caveat in this approach is a human error where one (me) forget to 
update the instruction
or forget to remove any additional flags used in the script temporarily 
- a mismatch
of the results could occur.

 From the next build (b120), I am going to update "2.1 Running tests in 
jdk/test" to

$ jtreg -dir:{openjdk source top directory}/jdk/test -verbose:summary 
-exclude:{openjdk source top directory}/jdk/test/ProblemList.txt 
-conc:auto -a -ignore:quiet -timeoutFactor:5 -agentvm -testjdk:{location 
of the test jdk} :jdk_core :jdk_svc :jdk_beans :jdk_imageio :jdk_sound 
:jdk_sctp javax/accessibility com/sun/java/swing javax/print sun/pisces 
com/sun/awt


> Alternatively, that segment of the script could be a candidate for a 
> target in
> one or more test/Makefile files.

This is good idea, but my experience with the 'make' is that if one 
target critically fail, all
subsequent targets will not run. I thought it is a restriction of 'make'.


Thanks
Balchandra

>
> -- Jon
>
>
> On 12/11/2013 08:58 AM, Balchandra Vaidya wrote:
>>
>> Hi Volker,
>>
>> The custom script I use just wrap the instruction posted (except
>> building jtreg). It just clones the openjdk repository, calls jtreg 
>> commands
>> (one each for jdk, langtools and hotspot) with those recommended
>> options, runs diffs,  runs nashorn tests, and archives results.
>>
>> Such convenient script may not fit under openjdk repo, but think of
>> adding it in code-tools project in the future.
>>
>>
>> Thanks
>> Balchandra
>>
>> On 11/12/2013 13:45, Volker Simonis wrote:
>>> Hi Balchandra,
>>>
>>> what do you think about checking in a little script or Makefile target
>>> which produces the test results you report for the ea builds?
>>>
>>> This would be of great help to produce comparable test results.
>>>
>>> Regards,
>>> Volker
>>>
>>> On Tue, Dec 10, 2013 at 5:39 PM, Balchandra Vaidya
>>> <balchandra.vaidya at oracle.com> wrote:
>>>> On 12/10/13 03:25 PM, Alan Bateman wrote:
>>>>> On 10/12/2013 11:20, Balchandra Vaidya wrote:
>>>>>> :
>>>>>>
>>>>>> 2) Testcase do not run with -agentvm option. It might be an
>>>>>>      issue in the testcase (or how the testcase was run). I will 
>>>>>> look into
>>>>>> it.
>>>>>>
>>>>>>      I missed this issue because I had changed my scripts to use 
>>>>>> -othervm
>>>>>>      sometime back for debugging/analyzing some testcase
>>>>>>      failure but forgot to change it back. Thank you for pointing 
>>>>>> it out.
>>>>> For Volker's benefit, othervm is where jtreg spins up a new VM for 
>>>>> each
>>>>> test, agentvm is where jtreg re-uses the VM if possible. There are 
>>>>> tests
>>>>> that don't or can't clean up and those tests need to ensure that 
>>>>> their @run
>>>>> tests have the /othervm option so that the tests runs in its own VM.
>>>>> Alternatively, the TEST.ROOT file has a key that lists directories 
>>>>> where all
>>>>> the tests in those directory trees must run in their own VM.
>>>>>
>>>>> Balchandra - one thing that be useful is to look at the client 
>>>>> tests to
>>>>> see what can run in agentvm and what can't. For the jdk_core and 
>>>>> jdk_svc
>>>>> tests then they can be run in either (although we're still picking 
>>>>> off
>>>>> issues as they found in agentvm mode). I'm not aware of any work 
>>>>> done yet on
>>>>> getting the tests in the jdk_desktop test group to run in agentvm 
>>>>> mode. It
>>>>> may be that TEST.ROOT needs to be updated to see a few additional 
>>>>> top-level
>>>>> directories so that all the tests in these areas temporarily run 
>>>>> in othervm
>>>>> mode.
>>>>
>>>> In the subset of the tests I have been running, it appears only one 
>>>> test [1]
>>>> is failing with agentvm option. I will try to run other tests in 
>>>> client
>>>> groups [Note: I will update the instruction to use jtreg groups 
>>>> with b119
>>>> results posting] and see what tests can be run in agentvm mode, but 
>>>> main
>>>> issue
>>>> in those remaining client tests are some tests may not consistently 
>>>> produce
>>>> same results - could stabilizing those tests be a goal for jdk9 repo ?
>>>>
>>>>
>>>> Thanks
>>>> Balchandra
>>>>
>>>> [1] com/sun/java/swing/plaf/windows/8016551/bug8016551.java
>>>>
>>>>> -Alan
>>>>
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/quality-discuss/attachments/20131211/c040d3ce/attachment-0001.html