RFR: Filing bug, ProblemListing, Backing out [v6]

[Igor Ignatyev]

> On Jul 14, 2020, at 8:12 AM, Alan Bateman <Alan.Bateman at oracle.com> wrote:
> 
> On 14/07/2020 15:54, Igor Ignatyev wrote:
>> :
>> `@ignore` is still used, albeit not very often, in hotspot tests for cases similar to the ones described here; given we
>> are seldomly running ProblemList-ed tests, it would be really unfortunate to have include tests which remove
>> `/etc/shadow` in such runs.
>> 
> A test that attempts to remove /etc/shadow sounds like a dangerous test. I remember there was one test in the libs areas but it has been fixed many years ago. I looked through the handful of tests that still have @ignore and the reasons aren't clear. Do you have examples of dangerous or other tests that aren't appropriate to put onto the jtreg exclude list? Only asking because it would be a lot simpler for everyone if there is one way to exclude tests.
> 
> -Alan

Luckily, there is nothing as dangerous as removal of '/etc/shadow' left in our test suites now ;). There are, however, ~70 tests marked @ignore k/w, almost equally distributed among core-libs (40%), hotspot (30%), and langtools (30%) test suites. I looked thru the number of @ignore-d tests and, although, I haven't found any dangerous tests, there are some tests whose execution is completely useless or impossible till the associated bug gets addressed, e.g. 5 mlvm tests @ignore-d due to 8194951. And as you mentioned, some tests were excluded due to unclear reasons, so I would be very reluctant to use ProblemList instead of @ignore k/w for them. I'm also aware of tests that are in a ProblemList when they were to be @ignore-d, e.g. compiler/c2/Test6852078.java. I do agree that having two ways to achieve _nearly_ identical behavior is confusing. With that being said, I still think we need to have a way to exclude useless and/or harmful tests explicitly, so they don't get executed in our runs of ProblemList-ed tests. Hopefully, this guide will make it less confusing for people, and the right exclusion way (which in most cases is ProblemList) is going to be used more often.

PS  I'll embark on the quest for "canonicalization" of existed usages of problem-list and @ignore in all test suites, so the code doesn't contradict the guide.

-- Igor