stable subset of jtreg tests?

[joe darcy]

Hi Martin,

On 9/16/2015 7:36 PM, Martin Buchholz wrote:
> OK, thanks, I managed to run my first tier test.
>
> What I (sometimes) really want is to apply a test filter to my test set:
> - those tests that are non-flaky and don't take "too many" resources 
> to run
> You could harvest both of those qualities from your many quality 
> engineering test runs.

FWIW, we see reasonable run times (20 minutes or less) on many platforms 
for running each of tier 1 and tier 2 tests if run on a well-sized 
server when using jtreg's agentvm mode and concurrency options.

-Joe

>
>
> On Wed, Sep 16, 2015 at 6:32 PM, Joseph D. Darcy <joe.darcy at oracle.com 
> <mailto:joe.darcy at oracle.com>> wrote:
>
>     Hi Martin,
>
>     On 9/16/2015 6:17 PM, Martin Buchholz wrote:
>>     Thanks Joe.
>>
>>     I see those tier definitions in the various TEST.groups files but
>>     it's not obvious to me how to run them all with one simple
>>     invocation (neither make nor jtreg).
>
>     Make support is coming, but not there yet:
>
>         JDK-8075571: Support tiered testing make targets
>
>>
>>     The command line you gave doesn't seem to tell jtreg which
>>     TEST.ROOT directories to operate on.
>>
>>     It looks like
>>     jtreg ... langtools/test:tier1 jdk/test:tier1
>>     works, but I was looking for a simpler way to "run all the tier1
>>     tests" without enumerating all the roots (there are currently 7!)
>
>     That is the simple(st) way currently.
>
>     The jdk, langtools, nashorn, and jaxp repos currently use the
>     tiered testing approach as defined in jtreg groups so you would
>     "only" need four roots.
>
>     The tier 2 tests are pretty stable too, but we certainly have
>     instances were new repeatable failures are introduced, but we're
>     getting better at addressing them quickly :-)
>
>     -Joe
>
>>
>>
>>     On Wed, Sep 16, 2015 at 5:25 PM, Joseph D. Darcy
>>     <joe.darcy at oracle.com <mailto:joe.darcy at oracle.com>> wrote:
>>
>>         Dropping the jtreg alias.
>>
>>
>>         On 9/16/2015 5:18 PM, Martin Buchholz wrote:
>>
>>             Just now for the first time I ran all the jtreg tests in
>>             the "jdk_stable" test set using
>>             make test TEST=jdk_stable
>>             and I got around 50 failures out of 5000.
>>             Is that expected?
>>
>>             Instead, I expect that with all the work on the test
>>             suite y'all have been doing, that the expected failure
>>             rate of something with a name like "jdk_stable" would be 0.
>>
>>
>>         For JDK 9 dev, the tier 1 tests across repos should be
>>         stable. Currently, only the langtools and jdk repos have
>>         non-empty tier 1 tests sets defined.
>>
>>         If you run
>>
>>             jtreg ... -exclude:ProblemList.txt -k:intermittent :tier1
>>
>>         there are some tier 1 test that are known to fail
>>         intermittently, but at this point with very low frequency (at
>>         least with the testing I'm exposed to).
>>
>>         HTH,
>>
>>         -Joe
>>
>>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/quality-discuss/attachments/20150917/80824d60/attachment.html>