JDK 8 b118 ea test results are now available

[Jonathan Gibbons]

Balchandra,

Thanks for posting the commands. That gives us something to reason about.

For my part, I have found -conc:auto to be too generous.   "auto" gets 
evaluated to the number of processors on the host system as seen by the 
Java runtime.  In my usage (on langtools) I have found a good rule of 
thumb to be half that number, so on a 32-way system, I use -conc:16, 
etc.    It is also a function of available memory.

I played with the idea of supporting expressions using auto, such as 
-conc:auto/2 or -conc:auto-1 -- but then I came to my senses. This 
calculation is much better done in the environment used to run jtreg.

I notice in your commands, you do NOT limit the amount of memory for 
each JVM with -Xmx.  That leaves all your JVMs using default ergonomics, 
which may be unduly optimistic when you have lots of JVMs running.

As a general rule for anyone using -conc to run tests concurrently, I 
strongly recommend that the first few times you do this, use your system 
activity profiling tool to monitor CPU and memory usage. If all your 
CPUs are redlining it at 100%, or if you start memory swapping, then 
your concurrency number is too high.  I would recommend that you target 
an average CPU utilization just below 100%, and no memory swapping.

A couple more comments inline.

-- Jon

P.S.   I'll see about converting some of these notes and guidelines into 
another jtreg page on OpenJDK.

-- Jon

On 12/11/2013 11:26 AM, Balchandra Vaidya wrote:
>
> On 11/12/2013 17:55, Jonathan Gibbons wrote:
>> Balchandra,
>>
>> The important part of that of interest here is the part about "calls 
>> jtreg commands
>> (one each for jdk, langtools and hotspot) with those recommended
>> options".
>>
>> Is there a way you could post here the segment of the script that 
>> does that?
>>
> Here it is
>
> 2.1 Running tests in jdk/test
> $ jtreg -dir:{openjdk source top directory}/jdk/test -verbose:summary 
> -exclude:{openjdk source top directory}/jdk/test/ProblemList.txt 
> -conc:auto -a -ignore:quiet -timeoutFactor:5 -othervm 
> -testjdk:{location of the test jdk} `cat dir.list 
> <http://download.java.net/jdk8/testresults/docs/dir.list>`
>
> 2.2 Running tests in langtools/test
> $ jtreg -dir:{openjdk source top directory}/langtools/test 
> -verbose:summary -conc:auto -a -ignore:quiet -timeoutFactor:5 -agentvm 
> -testjdk:{location of the test jdk} com tools
>
> 2.3 Running tests in hotspot/test
> $ jtreg -dir:{openjdk source top directory}/hotspot/test 
> -verbose:summary -conc:auto -a -ignore:quiet -timeoutFactor:5 -agentvm 
> -testjdk:{location of the test jdk} compiler gc runtime sanity 
> serviceability
>
>
> The instructions is at 
> http://download.java.net/jdk8/testresults/docs/howtoruntests.html
> and is linked from 
> http://www.java.net/download/jdk8/testresults/testresults.html
>
> The caveat in this approach is a human error where one (me) forget to 
> update the instruction
> or forget to remove any additional flags used in the script 
> temporarily - a mismatch
> of the results could occur.
>
> From the next build (b120), I am going to update "2.1 Running tests in 
> jdk/test" to
>
> $ jtreg -dir:{openjdk source top directory}/jdk/test -verbose:summary 
> -exclude:{openjdk source top directory}/jdk/test/ProblemList.txt 
> -conc:auto -a -ignore:quiet -timeoutFactor:5 -agentvm 
> -testjdk:{location of the test jdk} :jdk_core :jdk_svc :jdk_beans 
> :jdk_imageio :jdk_sound :jdk_sctp javax/accessibility 
> com/sun/java/swing javax/print sun/pisces com/sun/awt

Generally, I would say that if you need to put a list of groups on the 
command line, you need to update the groups file to create a new group 
that is composed of the individual groups you want to run.

I think it would be good to have a group such as :jdk_default or 
something like that.



>
>
>> Alternatively, that segment of the script could be a candidate for a 
>> target in
>> one or more test/Makefile files.
>
> This is good idea, but my experience with the 'make' is that if one 
> target critically fail, all
> subsequent targets will not run. I thought it is a restriction of 'make'.

Well, ...
a)  "make -k" keeps on going
b)  as a user of the Makefile, you should be looking for a single 
target, such as "make test", and it is up to the implementation of the 
target to run all the tests, even though some might fail. There are 
various ways to do this: for example, you can prefix a make rule with 
"-" to ignore the exit code from a command, or you can do a composite 
command like
     jtreg <opts> || echo some jtreg tests failed

>
>
> Thanks
> Balchandra
>
>>
>> -- Jon

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/quality-discuss/attachments/20131211/eb7d95a1/attachment.html