stable subset of jtreg tests?

[Martin Buchholz]

OK, thanks, I managed to run my first tier test.

What I (sometimes) really want is to apply a test filter to my test set:
- those tests that are non-flaky and don't take "too many" resources to run
You could harvest both of those qualities from your many quality
engineering test runs.


On Wed, Sep 16, 2015 at 6:32 PM, Joseph D. Darcy <joe.darcy at oracle.com>
wrote:

> Hi Martin,
>
> On 9/16/2015 6:17 PM, Martin Buchholz wrote:
>
> Thanks Joe.
>
> I see those tier definitions in the various TEST.groups files but it's not
> obvious to me how to run them all with one simple invocation (neither make
> nor jtreg).
>
>
> Make support is coming, but not there yet:
>
>     JDK-8075571: Support tiered testing make targets
>
>
> The command line you gave doesn't seem to tell jtreg which TEST.ROOT
> directories to operate on.
>
> It looks like
> jtreg ... langtools/test:tier1 jdk/test:tier1
> works, but I was looking for a simpler way to "run all the tier1 tests"
> without enumerating all the roots (there are currently 7!)
>
>
> That is the simple(st) way currently.
>
> The jdk, langtools, nashorn, and jaxp repos currently use the tiered
> testing approach as defined in jtreg groups so you would "only" need four
> roots.
>
> The tier 2 tests are pretty stable too, but we certainly have instances
> were new repeatable failures are introduced, but we're getting better at
> addressing them quickly :-)
>
> -Joe
>
>
>
> On Wed, Sep 16, 2015 at 5:25 PM, Joseph D. Darcy <joe.darcy at oracle.com>
> wrote:
>
>> Dropping the jtreg alias.
>>
>>
>> On 9/16/2015 5:18 PM, Martin Buchholz wrote:
>>
>>> Just now for the first time I ran all the jtreg tests in the
>>> "jdk_stable" test set using
>>> make test TEST=jdk_stable
>>> and I got around 50 failures out of 5000.
>>> Is that expected?
>>>
>>> Instead, I expect that with all the work on the test suite y'all have
>>> been doing, that the expected failure rate of something with a name like
>>> "jdk_stable" would be 0.
>>>
>>>
>> For JDK 9 dev, the tier 1 tests across repos should be stable. Currently,
>> only the langtools and jdk repos have non-empty tier 1 tests sets defined.
>>
>> If you run
>>
>>     jtreg ... -exclude:ProblemList.txt -k:intermittent :tier1
>>
>> there are some tier 1 test that are known to fail intermittently, but at
>> this point with very low frequency (at least with the testing I'm exposed
>> to).
>>
>> HTH,
>>
>> -Joe
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/quality-discuss/attachments/20150916/1becadf1/attachment-0001.html>