RFR: 7903785: Improve reporting for skipped tests [v2]

[Jonathan Gibbons]

On Mon, 19 Aug 2024 20:49:36 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

>> This is to support the ability to track `skipped` tests in the JDK `make test` system.
>> 
>> It is somewhere between "proof of concept", "draft PR", and initial version of a real PR.
>> 
>> The primary goal is to post the number of `skipped` tests in a backwards compatible way that does not break existing makefile rules or require any changes.  The makefiles currently have some "fuzzy parsing" to search for named values that may appear on the `Test results:` line near the end of the `jtreg` output to the console.   This PR _adds_ a new value to that line _without changing any existing values_. In particular, the `passed` value is unchanged and continues to include the `skipped` tests, as before, but for readers and tools alike, the number of skipped tests is also reported.
>> 
>> The PR also exposes a previously undocumented feature, to control the formatting of that line, using a format string that can be given to `jtreg` in a system property. The set of format specifiers in that string has been extended to include format specifiers for "skipped tests" and "passed tests, excluding skipped tests".
>> 
>> In exposing that feature, some design questions come to mind:
>> * the terminology of "ignored" tests is somewhat confusing -- it is about tests that were not selected by virtue of the keyword filter (and related `-k` option)
>> * if the number of `skipped` tests is interesting, there are other similar numbers that might also be interesting, such as the number of tests that were filtered out by a `@requires` tag -- which is similar to but more powerful than the keyword filter. But these extra numbers are contained within a composite filter, and may not have been so easily accessible when `jtreg` was built using older versions of JDK -- that is, before covariant returns.
>> * the `conditional space` and `conditional comma` provided as format specifiers do not work as well as intended if there is a plain-text label at the beginning of the format string.
>> 
>> But to summarize, this feature is primarily about reporting the number of `skipped` tests. We can either defer supporting additional values until a subsequent PR, or put more work into this PR to expose more of these "interesting" numbers.
>
> Jonathan Gibbons has updated the pull request incrementally with two additional commits since the last revision:
> 
>  - fix whitespace
>  - update with improved reporting for all not-run tests

Big new update.

In terms the original request, to report "skipped" tests when running tests with `make test`, there is almost no need to do anything, since the information is already available, if you know where to look. That being said, it is still a convenience to provide the information, to benefit anyone running `jtreg` directly, instead of via the makefiles.

But, in reviewing the conversations that led to this work, the underlying interest was more in terms of knowing and understanding the lack of test coverage in an area, which includes the tests that were not run at all, as well as just those that were started and then skipped.

So, this update provides detailed information on all the tests that were not run, as well as the tests that were run (and possibly skipped). The tests that are not run are those that were specified in the "initial files" (including groups), but which were subsequently filtered out by one of the various filters.

I gave up trying to find meaningful single words that describe the reasons that a test might be filtered out, and went with short phrases instead.  In addition, the `jtreg` FAQ is updated which more detailed explanations, to help anyone who is search for any such details.

In terms of formatting the numbers, I chose to keep all the labeled numbers on a single line, just because there is a spectrum of conditions here that got in the way of splitting the info into many lines: _tests that were run_, _tests that were run but skipped_, _tests that were not run_.

*In addition,*
A new file `text/notRun.txt` is generated in the report directory that uses the same format as `text/summary.txt` but which lists the primary reason why each test was not executed -- where _primary_ means _first_ in the unspecified order in which the filters are applied.  For the record, I tried quite hard to include the information in `summary.txt` but although I succeeded at one point, the results were somewhat unsatisfactory, not least because it meant overwriting and results for a test in the work directory.

*Notes,*
1. I (re)discovered the `jtreg` `-report` option, which can be used to control the set of tests in the report that is generated at the end of a test run. In conjunction with the effectively undocumented support for `JT Harness` preferences, It is possible to have the system report all the filters that rejected any individual test, not just the first one. But the filters are identified by names that are neither consistent nor "user-friendly".
2. In documenting the use of the keywords filter, I realize that we do not seem to adequately document the "implicit" keywords defined by any test description, not the "effective" keyword expression created from the `jtreg` command-line options. But that is another RFE, for another day.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/217#issuecomment-2297460063