RFR: 7903785: Improve reporting for skipped tests [v3]

[Jonathan Gibbons]

On Fri, 30 Aug 2024 11:37:19 GMT, Ivan Bereziuk <duke at openjdk.org> wrote:

> Would it be possible to generate a report for every test which was skipped? I am not sure if that's already possible or will be possible with this change. It's likely out of scope. But it would be nice to know your thoughts.

I don't know what that means, or what you are asking for.

Partly to clarify, in this context, "skipped" means a test was started, but the test dynamically determined it could not proceed, and so threw a `jtreg.SkippedException`.  Because the test was started, there will be a `.jtr` file giving the standard details of how the test was compiled and executed, and will contain any output written by the test up to the point it threw the exception.

Separately, some tests are "not run" (in this context, that's not the same as "skipped").  A test may be not run for many reasons, but all generally of the form that the test was not accepted by one of the various filters, such as the keyword filter, the exclude-list filter, the `requires` filter, and so on. If a test is "not run" there is nothing additional that can reasonably be reported. Yes, in principle, `jtreg` could report fine-grain detail of how and why whichever filter(s) rejected the test, but architecturally, that is not really possible.

To summarize, the `notRun.txt` file will contain a brief one-line summary of why a test was not run. So, you will be able to see whether it was an issue with keywords, requires, modules, problem list, and so on. That is about the best we can do at this time.    We _could_, but do not, report _all_ the filters that reject any one test.  For now, it is "first one wins. The goal/emphasis is to indicate to a user which tests were not run, with it being of secondary importance _why_ it was not run.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/217#issuecomment-2339286178