RFR: Filing bug, ProblemListing, Backing out [v2]

[Jesper Wilhelmsson]

On Mon, 6 Jul 2020 19:08:20 GMT, Phil Race <prr at openjdk.org> wrote:

>> Jesper Wilhelmsson has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Fixed comments from review
>
> src/next.md line 22:
> 
>> 21: When a new failure is found in the JDK a bug should be filed to describe and track the issue. Depending on your
>> role in the OpenJDK you can either use the [Bug Report Tool](https://bugreport.java.com/) or, if you are Author in an
>> OpenJDK Project, report the bug directly in the [JDK Bug System](https://bugs.openjdk.java.net/). Try to make the bug
>> report as complete as possible to make it easier to triage and investigate the bug. It's not the reporter's
>> responsibility to set a correct priority, but a good guess is always better than a default value. To help with setting
>> the right priority consider things like how the bug impacts the product and our testing, how likely is it that the bug
>> triggers, and how difficult is it to work around the bug if it's not fixed soon. 22:  23: A few things to keep in mind
>> when filing a new bug:
> 
> and whether it is a recent regression, since that may break existing applications. regressions are often higher
> priority than long standing bugs.

Added this to the text.

> src/next.md line 42:
> 
>> 41: Sometimes tests break. It could be due to bugs in the test itself, or due to changed functionality in the code that
>> the test is testing. While working on a fix, it can be useful to stop the test from being executed in everyone else's
>> testing to reduce noise, especially if the test is expected to fail for more than a day. There are two ways to stop a
>> test from being run in standard test runs: ProblemListing and using the `@ignore` keyword. Removing tests isn't the
>> standard way to remove a failure. A failing test is expected to be a regression and should ideally be handled promptly
>> with high urgency. 42:  43: I'll say it right away so that it's not forgotten at the end: Remember to remove the JBS id
>> from the ProblemList or the test when the bug is closed. This is especially easy to forget if a bug is closed as a
>> duplicate or 'Will Not Fix'.
> 
> this isn't the whole picture A test may fail because it is now being tested in  new environment, (eg new platform or
> platform release) not just because of the afore stated reasons. This would not be a regression but might still be
> important.

Agreed. I have updated the text to reflect this.

> src/next.md line 48:
> 
>> 47: ProblemListing is used for a short term exclusion while a test is being fixed, and for the exclusion of
>> intermittently failing tests that cause too much noise, but can still be useful to run on an ad-hoc basis.
>> ProblemListing is done in the file `ProblemList.txt`. There are actually several ProblemList files to choose from.
>> Their location and name hint about what area or feature each file belongs to. Each file has sections for different
>> components. All ProblemList files complement each other to build the total set of tests to exclude in JTReg runs. 48:
>> 49: ~~~
> 
> the emphasis that the exclusion is short lived is sadly unduly optimistic. Problem listing can be used for long term
> exclusions too. It is still better than deleting a valid test.

"is" -> "should be" ?

-------------

PR: https://git.openjdk.java.net/guide/pull/21