<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi Martin,<br>
<br>
<div class="moz-cite-prefix">On 9/16/2015 7:36 PM, Martin Buchholz
wrote:<br>
</div>
<blockquote
cite="mid:CA+kOe0_7MOVS13=6rRn2n8P-TH8xXhfKBZ70=YZ=q=py-UfkzA@mail.gmail.com"
type="cite">
<div dir="ltr">OK, thanks, I managed to run my first tier test.
<div><br>
</div>
<div>What I (sometimes) really want is to apply a test filter to
my test set:</div>
<div>- those tests that are non-flaky and don't take "too many"
resources to run</div>
<div>You could harvest both of those qualities from your many
quality engineering test runs.<br>
</div>
</div>
</blockquote>
<br>
FWIW, we see reasonable run times (20 minutes or less) on many
platforms for running each of tier 1 and tier 2 tests if run on a
well-sized server when using jtreg's agentvm mode and concurrency
options.<br>
<br>
-Joe<br>
<br>
<blockquote
cite="mid:CA+kOe0_7MOVS13=6rRn2n8P-TH8xXhfKBZ70=YZ=q=py-UfkzA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Sep 16, 2015 at 6:32 PM, Joseph
D. Darcy <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:joe.darcy@oracle.com" target="_blank">joe.darcy@oracle.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Hi Martin,<span
class=""><br>
<br>
<div>On 9/16/2015 6:17 PM, Martin Buchholz wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Thanks Joe.
<div><br>
</div>
<div>I see those tier definitions in the various
TEST.groups files but it's not obvious to me how
to run them all with one simple invocation
(neither make nor jtreg).</div>
</div>
</blockquote>
<br>
</span> Make support is coming, but not there yet:<br>
<br>
JDK-8075571: Support tiered testing make targets <br>
<span class=""> <br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>The command line you gave doesn't seem to tell
jtreg which TEST.ROOT directories to operate on.</div>
<div><br>
</div>
<div>It looks like </div>
<div>jtreg ... langtools/test:tier1 jdk/test:tier1 </div>
<div>works, but I was looking for a simpler way to
"run all the tier1 tests" without enumerating all
the roots (there are currently 7!)</div>
</div>
</blockquote>
<br>
</span> That is the simple(st) way currently.<br>
<br>
The jdk, langtools, nashorn, and jaxp repos currently use
the tiered testing approach as defined in jtreg groups so
you would "only" need four roots.<br>
<br>
The tier 2 tests are pretty stable too, but we certainly
have instances were new repeatable failures are
introduced, but we're getting better at addressing them
quickly :-)<br>
<br>
-Joe<span class=""><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Sep 16, 2015 at
5:25 PM, Joseph D. Darcy <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:joe.darcy@oracle.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:joe.darcy@oracle.com">joe.darcy@oracle.com</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">Dropping the jtreg
alias.
<div>
<div><br>
<br>
On 9/16/2015 5:18 PM, Martin Buchholz wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex"> Just now for
the first time I ran all the jtreg tests
in the "jdk_stable" test set using<br>
make test TEST=jdk_stable<br>
and I got around 50 failures out of 5000.<br>
Is that expected?<br>
<br>
Instead, I expect that with all the work
on the test suite y'all have been doing,
that the expected failure rate of
something with a name like "jdk_stable"
would be 0.<br>
<br>
</blockquote>
<br>
</div>
</div>
For JDK 9 dev, the tier 1 tests across repos
should be stable. Currently, only the langtools
and jdk repos have non-empty tier 1 tests sets
defined.<br>
<br>
If you run<br>
<br>
jtreg ... -exclude:ProblemList.txt
-k:intermittent :tier1<br>
<br>
there are some tier 1 test that are known to
fail intermittently, but at this point with very
low frequency (at least with the testing I'm
exposed to).<br>
<br>
HTH,<br>
<br>
-Joe<br>
<br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</span></div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</body>
</html>