RFR(S) : 8078450 : Implement consistent process for quarantine of tests

[Dmitry Fazunenenko]

Hi Igor,

Thank you for doing that long-awaited change!
The fix looks good, but I have a few minor questions/comments:

1) Have you moved all the tests which are currently @ignored to the 
problem list or some of them intentionally left @ignored?

2) Copyrights

# Copyright (c) 2009, 2016, Oracle and/or its affiliates. All rights reserved.
-->
# Copyright (c) 2016, Oracle and/or its affiliates. All rights reserved.


3) Would you consider correction in wording:

# List of quarantined tests -- tests that should not be run by default, because
# they introduce noise in test results.
-->
# List of problematic tests -- tests that should not be run by default, because
# they may fail due to known reason. The reason (CR#) must be mandatory specified.

4) Would you consider to use the same formatting as jdk does:

compiler/codecache/stress/OverloadCompileQueueTest.java 8166554 generic-all
compiler/compilercontrol/jcmd/ClearDirectivesFileStackTest.java 8140405 generic-all
compiler/jvmci/compilerToVM/GetResolvedJavaTypeTest.java 8158860 generic-all
compiler/jvmci/compilerToVM/InvalidateInstalledCodeTest.java 8163894 generic-all
compiler/startup/SmallCodeCacheStartup.java 8134286 generic-all
compiler/tiered/LevelTransitionTest.java 8067651 generic-all
compiler/types/correctness/CorrectnessTest.java 8066173 generic-all
compiler/types/correctness/OffTest.java 8066173 generic-all
-->
compiler/codecache/stress/OverloadCompileQueueTest.java             8166554 generic-all

compiler/compilercontrol/jcmd/ClearDirectivesFileStackTest.java     8140405 generic-all

compiler/jvmci/compilerToVM/GetResolvedJavaTypeTest.java            8158860 generic-all

compiler/jvmci/compilerToVM/InvalidateInstalledCodeTest.java        8163894 generic-all

compiler/startup/SmallCodeCacheStartup.java                         8134286 generic-all

compiler/tiered/LevelTransitionTest.java                            8067651 generic-all

compiler/types/correctness/CorrectnessTest.java                     8066173 generic-all
compiler/types/correctness/OffTest.java                             8066173 generic-all


Thanks,
Dima

On 02.12.2016 22:27, Igor Ignatyev wrote:
> http://cr.openjdk.java.net/~iignatyev/8078450/webrev.00/
>> 115 lines changed: 97 ins; 17 del; 1 mod;
> Hi all,
>
> could you please review this small changeset which introduces ProblemList for quarantining hotspot test and adds all currently quarantined tests to it?
>
> a bit of background:
> there are two cases when one wants to remove a test from test execution:
>   - a test sporadically fails, but there is still a value in running this test, e.g. it can fail in some other way and reveal another problem within the product. we want to run such tests, but we don’t want to have noise from these tests. to get both, we are going to _quarantine_ these tests and run them separately.
>   - a test always fails or can break other tests or host. running such tests makes more harm than good, so they should be _excluded_ from test execution completely.
> jtreg provides two exclusion mechanisms: @ignore tag and ProblemList. ProblemList gives a possibility to remove a test from execution on a specific OS, arch, and since the expectation is to have more quarantined tests than excluded, it was decided to use ProblemList to quarantine and @ignore to exclude, another reason is to have consistent process for hotspot and jdk tests.
>
> in two words, @ignore to exclude, ProblemList to quarantine.
>
> webrev: http://cr.openjdk.java.net/~iignatyev/8078450/webrev.00/
> jbs: https://bugs.openjdk.java.net/browse/JDK-8078450
>
> Thanks,
> — Igor