RFR(S) : 8078450 : Implement consistent process for quarantine of tests

[Igor Ignatyev]

Hi David,

thank you for your review.

> Is that target used by JPRT when running jtreg tests? What about rbt etc?
all our test execution systems either use jtreg_tests target or check if ProblemList files exist add corresponding -exclude options explicitly.

> That needs a bit of a "heads up” email
sure, I’ll send an email w/ explanations how to run only stable hotspot tests.

Thanks,
— Igor

> On Dec 5, 2016, at 7:02 AM, David Holmes <david.holmes at oracle.com> wrote:
> 
> Hi Igor,
> 
> Seems okay on principle.
> 
> The previously undefined JTREG_EXTENSIONS is passed to jtreg via the Makefile jtreg_tests target. Is that target used by JPRT when running jtreg tests? What about rbt etc?
> 
> When running jtreg standalone/locally we now have to remember to add potentially two --exclude options. That needs a bit of a "heads up" email beyond this RFR in my opinion.
> 
> Nit: The new ProblemList.txt should have a single year, 2016, in its copyright header.
> 
> +# Copyright (c) 2009, 2016, Oracle and/or its affiliates. All rights reserved.
> 
> Thanks,
> David
> 
> On 3/12/2016 5:27 AM, Igor Ignatyev wrote:
>> http://cr.openjdk.java.net/~iignatyev/8078450/webrev.00/
>>> 115 lines changed: 97 ins; 17 del; 1 mod;
>> 
>> Hi all,
>> 
>> could you please review this small changeset which introduces ProblemList for quarantining hotspot test and adds all currently quarantined tests to it?
>> 
>> a bit of background:
>> there are two cases when one wants to remove a test from test execution:
>> - a test sporadically fails, but there is still a value in running this test, e.g. it can fail in some other way and reveal another problem within the product. we want to run such tests, but we don’t want to have noise from these tests. to get both, we are going to _quarantine_ these tests and run them separately.
>> - a test always fails or can break other tests or host. running such tests makes more harm than good, so they should be _excluded_ from test execution completely.
>> jtreg provides two exclusion mechanisms: @ignore tag and ProblemList. ProblemList gives a possibility to remove a test from execution on a specific OS, arch, and since the expectation is to have more quarantined tests than excluded, it was decided to use ProblemList to quarantine and @ignore to exclude, another reason is to have consistent process for hotspot and jdk tests.
>> 
>> in two words, @ignore to exclude, ProblemList to quarantine.
>> 
>> webrev: http://cr.openjdk.java.net/~iignatyev/8078450/webrev.00/
>> jbs: https://bugs.openjdk.java.net/browse/JDK-8078450
>> 
>> Thanks,
>> — Igor
>>