<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
<div class="moz-cite-prefix">On 12/11/13 07:50 PM, Jonathan Gibbons
wrote:<br>
</div>
<blockquote cite="mid:52A8C208.3080806@oracle.com" type="cite">
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
Balchandra,<br>
<br>
Thanks for posting the commands. That gives us something to reason
about.<br>
<br>
For my part, I have found -conc:auto to be too generous. "auto"
gets evaluated to the number of processors on the host system as
seen by the Java runtime. In my usage (on langtools) I have found
a good rule of thumb to be half that number, so on a 32-way
system, I use -conc:16, etc. It is also a function of available
memory.<br>
<br>
I played with the idea of supporting expressions using auto, such
as -conc:auto/2 or -conc:auto-1 -- but then I came to my senses.
This calculation is much better done in the environment used to
run jtreg.<br>
<br>
I notice in your commands, you do NOT limit the amount of memory
for each JVM with -Xmx. That leaves all your JVMs using default
ergonomics, which may be unduly optimistic when you have lots of
JVMs running.<br>
<br>
As a general rule for anyone using -conc to run tests
concurrently, I strongly recommend that the first few times you do
this, use your system activity profiling tool to monitor CPU and
memory usage. If all your CPUs are redlining it at 100%, or if
you start memory swapping, then your concurrency number is too
high. I would recommend that you target an average CPU
utilization just below 100%, and no memory swapping.<br>
<br>
</blockquote>
<br>
It is a good information.<br>
<br>
First, I usually run tests on 4 way and 8 way systems. I believe the<br>
objective here is to provide repeatable test results. But, the <br>
hardware availability, OS versions, kernel, patches, window manager,<br>
jtreg options, and many other factors play a role to produce <br>
consistent test results. The truth is we may not able to produce <br>
results using all combinations, but the results posted can
reasonably<br>
be used for a comparison. That is, the result posted is not<br>
everything/unbreakable.<br>
<br>
<br>
Is it fair to say something like "The test results posted here used<br>
-conc:2 -Xmx:512m ..... options. Result could vary if other options
<br>
were used. However, the community members are encouraged <br>
to send email to quality-discuss or file bugs if any issues found".
If<br>
a testcase pass with -conc:2 but fail with -conc:4 means there <br>
might be a bug in the testcase - such failures are good to know.<br>
<br>
<br>
Second point is to make test execution easier. Does it make sense<br>
to add default options (concurrency, memory etc.) to jtreg? Or does
<br>
it make sense to create a separate environment files (similar to <br>
what is used in JCK) and read those files in jtreg or other wrapper<br>
script? However, I also think that are such micro tuning necessary?
<br>
Testing different combinations may find bugs which is beneficial to
<br>
OpenJDK.<br>
<br>
<br>
<br>
<blockquote cite="mid:52A8C208.3080806@oracle.com" type="cite"> A
couple more comments inline.<br>
<br>
-- Jon<br>
<br>
P.S. I'll see about converting some of these notes and
guidelines into another jtreg page on OpenJDK.<br>
<br>
-- Jon<br>
<br>
<div class="moz-cite-prefix">On 12/11/2013 11:26 AM, Balchandra
Vaidya wrote:<br>
</div>
<blockquote cite="mid:52A8BC76.60409@oracle.com" type="cite">
<meta content="text/html; charset=UTF-8"
http-equiv="Content-Type">
<br>
<div class="moz-cite-prefix">On 11/12/2013 17:55, Jonathan
Gibbons wrote:<br>
</div>
<blockquote cite="mid:52A8A72C.7050300@oracle.com" type="cite">Balchandra,
<br>
<br>
The important part of that of interest here is the part about
"calls jtreg commands <br>
(one each for jdk, langtools and hotspot) with those
recommended <br>
options". <br>
<br>
Is there a way you could post here the segment of the script
that does that? <br>
<br>
</blockquote>
Here it is<br>
<br>
<meta http-equiv="content-type" content="text/html;
charset=UTF-8">
<table height="300" border="0" width="749">
<tbody>
<tr>
<td>2.1 Running tests in jdk/test </td>
</tr>
<tr>
<td>
<div style="margin-left: 40px;"> $ jtreg -dir:{openjdk
source top directory}/jdk/test -verbose:summary
-exclude:{openjdk source top
directory}/jdk/test/ProblemList.txt -conc:auto -a
-ignore:quiet -timeoutFactor:5 -othervm
-testjdk:{location of the test jdk} `cat <a
moz-do-not-send="true"
href="http://download.java.net/jdk8/testresults/docs/dir.list">dir.list</a>`
<br>
<br>
</div>
</td>
</tr>
<tr>
<td> 2.2 Running tests in langtools/test </td>
</tr>
<tr>
<td>
<div style="margin-left: 40px;"> $ jtreg -dir:{openjdk
source top directory}/langtools/test -verbose:summary
-conc:auto -a -ignore:quiet -timeoutFactor:5 -agentvm
-testjdk:{location of the test jdk} com tools <br>
<br>
</div>
</td>
</tr>
<tr>
<td> 2.3 Running tests in hotspot/test </td>
</tr>
<tr>
<td>
<div style="margin-left: 40px;"> $ jtreg -dir:{openjdk
source top directory}/hotspot/test -verbose:summary
-conc:auto -a -ignore:quiet -timeoutFactor:5 -agentvm
-testjdk:{location of the test jdk} compiler gc
runtime sanity serviceability </div>
</td>
</tr>
</tbody>
</table>
<br>
The instructions is at <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="http://download.java.net/jdk8/testresults/docs/howtoruntests.html">http://download.java.net/jdk8/testresults/docs/howtoruntests.html</a><br>
and is linked from <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="http://www.java.net/download/jdk8/testresults/testresults.html">http://www.java.net/download/jdk8/testresults/testresults.html</a><br>
<br>
The caveat in this approach is a human error where one (me)
forget to update the instruction<br>
or forget to remove any additional flags used in the script
temporarily - a mismatch<br>
of the results could occur. <br>
<br>
From the next build (b120), I am going to update "2.1 Running
tests in jdk/test" to<br>
<br>
<meta http-equiv="content-type" content="text/html;
charset=UTF-8">
$ jtreg -dir:{openjdk source top directory}/jdk/test
-verbose:summary -exclude:{openjdk source top
directory}/jdk/test/ProblemList.txt -conc:auto -a -ignore:quiet
-timeoutFactor:5 -agentvm -testjdk:{location of the test jdk}
:jdk_core :jdk_svc :jdk_beans :jdk_imageio :jdk_sound :jdk_sctp
javax/accessibility com/sun/java/swing javax/print sun/pisces
com/sun/awt <br>
</blockquote>
<br>
Generally, I would say that if you need to put a list of groups on
the command line, you need to update the groups file to create a
new group that is composed of the individual groups you want to
run.<br>
<br>
I think it would be good to have a group such as :jdk_default or
something like that.<br>
<br>
<br>
</blockquote>
<br>
Yes, it is a good idea.<br>
<br>
<blockquote cite="mid:52A8C208.3080806@oracle.com" type="cite"> <br>
<blockquote cite="mid:52A8BC76.60409@oracle.com" type="cite"> <br>
<br>
<blockquote cite="mid:52A8A72C.7050300@oracle.com" type="cite">Alternatively,
that segment of the script could be a candidate for a target
in <br>
one or more test/Makefile files. <br>
</blockquote>
<br>
This is good idea, but my experience with the 'make' is that if
one target critically fail, all<br>
subsequent targets will not run. I thought it is a restriction
of 'make'.<br>
</blockquote>
<br>
Well, ...<br>
a) "make -k" keeps on going<br>
b) as a user of the Makefile, you should be looking for a single
target, such as "make test", and it is up to the implementation of
the target to run all the tests, even though some might fail.
There are various ways to do this: for example, you can prefix a
make rule with "-" to ignore the exit code from a command, or you
can do a composite command like<br>
jtreg <opts> || echo some jtreg tests failed<br>
</blockquote>
<br>
I will try again. I remember "make -k" didn't work from start to
end. That<br>
was an year ago, so I will retry. Is it time to say goodbye to
'make' and say<br>
hello to 'ant/gradle' for jdk9 ?<br>
<br>
<br>
Thanks<br>
Balchandra<br>
<br>
<br>
<blockquote cite="mid:52A8C208.3080806@oracle.com" type="cite"> <br>
<blockquote cite="mid:52A8BC76.60409@oracle.com" type="cite"> <br>
<br>
Thanks<br>
Balchandra<br>
<br>
<blockquote cite="mid:52A8A72C.7050300@oracle.com" type="cite">
<br>
-- Jon <br>
</blockquote>
</blockquote>
<br>
</blockquote>
<br>
</body>
</html>