[heads-up] changes in quarantine/exclusion of hotspot jtreg tests

[Igor Ignatyev]

it seems I made a typo in -ignore syntax. in all occurrences it should be '-ignore:quiet', not '-ignore:quite’. thanks for Dan Daugherty who noticed that.

— Igor

> On Dec 7, 2016, at 6:17 PM, Igor Ignatyev <igor.ignatyev at oracle.com> wrote:
> 
> Hi everyone,
> 
> I would like to give you a heads-up about the recent changes[1] in how we quarantine and exclude jtreg tests in hotspot and possible impact of these changes.
> 
> in two words:
> 1. To quarantine a hotspot jtreg test please update ProblemList, do not use @ignore (use @ignore only to never execute a test until the problem is fixed).
> 2. If you use RBT or 'make test', you do not need to change anything.
> 3. If you use jtreg directly, please add '-exclude:$WS/hotspot/test/ProblemList.txt -exclude:$WS/hotspot/test/closed/ProblemList.txt' not to run quarantined tests.
> 
> in more details:
> The current processes enact us to exclude a failing test from regular execution if we can not fix the problem which causes the test to fail. As you may know, we differentiate two types of failing tests basing on the value of their running: "quarantined" and “excluded". A quarantined test is a test which sporadically fails, but there is still a value in running this test. An excluded test is a test which always fails or can break other tests or execution environment. The changes[1] unify handling failing jdk and hotspot jtreg tests: quarantined tests should be put into ProblemList.txt, excluded tests should be marked by @ignore. You can find more details on that decision in bug descriptions[1-2] and RFR[3].
> So as of today, a quarantined hotspot jtreg test must be put to ProblemList, please stop using @ignore for that.
> 
> No matter which exclusion mechanism is used, id of the bug causing a problem must be provided. This must be the bug which will fix the problem, not the one used to remove a test from execution. In case of excluded tests, one must specify a space separated list of bug ids after @ignore tag in test's description[4]. In ProblemList.txt, a quarantined testname(test filename) must be followed by a comma separated list of bug ids[5]. The one who fixes a bug is responsible for cleaning ProblemList/@ignore.
> 
> The makefiles and scripts are adjusted to the fix, therefore for RBT, JPRT users and those who run tests by 'make test’ these changes are absolutely transparent, neither quarantine nor excluded tests will be run by default. The changes affect only those who run tests using jtreg directly. To skip running quarantined tests, which now listed in ProblemList.txt, an extra '-exclude:<full-path-to-ProblemList.txt>' jtreg flag should be specified, if there is more than one ProblemList, e.g. open and closed, one should specify '-exclude' option several times. To skip running excluded tests, '-ignore:quite' flag should be given (otherwise such tests will be reported as errors).
> 
> The following are examples for three main use-cases ($WS is a full path to the root of workspace):
> 1. run only stable tests (skip quarantined and excluded tests): 
> $ jtreg -dir:$WS/hotspot/test -ignore:quite -exclude:$WS/hotspot/test/ProblemList.txt -exclude:$WS/hotspot/test/closed/ProblemList.txt :hotspot_all
> or if you do not have closed part:
> $ jtreg -dir:$WS/hotspot/test -ignore:quite -exclude:$WS/hotspot/test/ProblemList.txt :hotspot_all
> 2. run stable and quarantined tests:
> $ jtreg -dir:$WS/hotspot/test -ignore:quite :hotspot_all
> 3. run all tests:
> $ jtreg -dir:$WS/hotspot/test -ignore:run :hotspot_all
> 
> Thanks,
> — Igor
> 
> [1] https://bugs.openjdk.java.net/browse/JDK-8078450
> [2] https://bugs.openjdk.java.net/browse/JDK-8145079
> [3] http://mail.openjdk.java.net/pipermail/hotspot-dev/2016-December/025436.html
> [4] test/bar/TestFoo.java:
> /**
> * @test TestFoo
> * @ignore 1234567 2345678
> */
> [5] test/ProblemList.txt:
> test/bar/TestBaz.java 7654321,8765432 generic-all
>