stable subset of jtreg tests?

[Joseph D. Darcy]

Hi Martin,

On 9/16/2015 6:17 PM, Martin Buchholz wrote:
> Thanks Joe.
>
> I see those tier definitions in the various TEST.groups files but it's 
> not obvious to me how to run them all with one simple invocation 
> (neither make nor jtreg).

Make support is coming, but not there yet:

     JDK-8075571: Support tiered testing make targets

>
> The command line you gave doesn't seem to tell jtreg which TEST.ROOT 
> directories to operate on.
>
> It looks like
> jtreg ... langtools/test:tier1 jdk/test:tier1
> works, but I was looking for a simpler way to "run all the tier1 
> tests" without enumerating all the roots (there are currently 7!)

That is the simple(st) way currently.

The jdk, langtools, nashorn, and jaxp repos currently use the tiered 
testing approach as defined in jtreg groups so you would "only" need 
four roots.

The tier 2 tests are pretty stable too, but we certainly have instances 
were new repeatable failures are introduced, but we're getting better at 
addressing them quickly :-)

-Joe

>
>
> On Wed, Sep 16, 2015 at 5:25 PM, Joseph D. Darcy <joe.darcy at oracle.com 
> <mailto:joe.darcy at oracle.com>> wrote:
>
>     Dropping the jtreg alias.
>
>
>     On 9/16/2015 5:18 PM, Martin Buchholz wrote:
>
>         Just now for the first time I ran all the jtreg tests in the
>         "jdk_stable" test set using
>         make test TEST=jdk_stable
>         and I got around 50 failures out of 5000.
>         Is that expected?
>
>         Instead, I expect that with all the work on the test suite
>         y'all have been doing, that the expected failure rate of
>         something with a name like "jdk_stable" would be 0.
>
>
>     For JDK 9 dev, the tier 1 tests across repos should be stable.
>     Currently, only the langtools and jdk repos have non-empty tier 1
>     tests sets defined.
>
>     If you run
>
>         jtreg ... -exclude:ProblemList.txt -k:intermittent :tier1
>
>     there are some tier 1 test that are known to fail intermittently,
>     but at this point with very low frequency (at least with the
>     testing I'm exposed to).
>
>     HTH,
>
>     -Joe
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/quality-discuss/attachments/20150916/ff58c10c/attachment.html>