RFR: 7903785: Improve reporting for skipped tests

[Jonathan Gibbons]

On Wed, 7 Aug 2024 18:25:13 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

> This is to support the ability to track `skipped` tests in the JDK `make test` system.
> 
> It is somewhere between "proof of concept", "draft PR", and initial version of a real PR.
> 
> The primary goal is to post the number of `skipped` tests in a backwards compatible way that does not break existing makefile rules or require any changes.  The makefiles currently have some "fuzzy parsing" to search for named values that may appear on the `Test results:` line near the end of the `jtreg` output to the console.   This PR _adds_ a new value to that line _without changing any existing values_. In particular, the `passed` value is unchanged and continues to include the `skipped` tests, as before, but for readers and tools alike, the number of skipped tests is also reported.
> 
> The PR also exposes a previously undocumented feature, to control the formatting of that line, using a format string that can be given to `jtreg` in a system property. The set of format specifiers in that string has been extended to include format specifiers for "skipped tests" and "passed tests, excluding skipped tests".
> 
> In exposing that feature, some design questions come to mind:
> * the terminology of "ignored" tests is somewhat confusing -- it is about tests that were not selected by virtue of the keyword filter (and related `-k` option)
> * if the number of `skipped` tests is interesting, there are other similar numbers that might also be interesting, such as the number of tests that were filtered out by a `@requires` tag -- which is similar to but more powerful than the keyword filter. But these extra numbers are contained within a composite filter, and may not have been so easily accessible when `jtreg` was built using older versions of JDK -- that is, before covariant returns.
> * the `conditional space` and `conditional comma` provided as format specifiers do not work as well as intended if there is a plain-text label at the beginning of the format string.
> 
> But to summarize, this feature is primarily about reporting the number of `skipped` tests. We can either defer supporting additional values until a subsequent PR, or put more work into this PR to expose more of these "interesting" numbers.

I've been thinking about this some more, and I'm not sure how much we need 
the work as-is, and/or what the overall desired functionality should be.

Tests may be selected for execution in a number of ways; equivalently,
they may be filtered out from selection in the same number of ways.
Here are the ways that tests can be selected or filtered out.

* The initial set of tests to be executed is given by the set of 
  initial files or groups, given as paths on the command-line.

* The tests are then passed through a series of filters to
  select or reject tests to be executed. Most of these are
  relatively simple filters, specified by command-line options.
  The order in which these filters are applied is not specified.
  A test must be accepted by all these filters to be executed;
  otherwise, it is "first one wins" to reject a test if it should
  not be executed.

  * _exclude list_ -- exclude tests given in "problem-list" files,
    given with `-exclude` on the command line
  * _keywords_ -- only run tests matching a keyword expression 
    given with `-k` on the command line
  * _prior status_ -- only run tests whose prior execution status 
    matches those given on the command line with `-status`
  * _time limit_ -- only run those tests that are expected to take 
    less time than a value given on the command line with `-tl`
    or `-timelimit`
  * _match list_ -- only tests given in "problem-list" files,
    given with `-match` on the command line; used to explicitly
    run tests that are normally excluded

The last filter in this group is always enabled and is not
directly controlled by command-line options.

* _requires_ -- check whether the test statically meets the
  conditions met in a `@requires` tag in the test description.
  This may use characteristics (properties) of the test VM, 
  such as the OS name, and family, the max memory size and 
  so on. Some of the properties may be exported from the
  underlying VM.  The expression is evaluated by `jtreg`;
  the test itself is not executed at this stage.

If a test given by one of the initial files or test groups
is accepted by all the preceding filters, it will be executed.
When it is executed, there are various possible outcomes.

* The test may dynamically determine that it cannot be executed
  as intended. If this occurs, it should indicate that outcome
  by throwing `jtreg.SkippedException`, which will be treated 
  by the harness as a "skip": not passed but not failed either".
  This can be thought of as a "dynamic" filter, similar to the
  static "requires" filter. The difference is that the check
  happens within the running test.

* A combo or framework test may determine that some parts of the 
  cannot be excuted as intended. If this occurs, the framework
  should report that in some appropriate manner, and then
  report passed or failed for the parts that it can execute.

* The test should report passed or failed as appropriate.

So, where does that leave this work.

1. Because skipped tests (those that throw `jtreg.SkippedException`)
   are actually executed, the information about those tests
   are recorded in the exeisting `report/text/summary.txt` file.
   Look for the lines containing `Passed. Skipped: jtreg.SkippedException:`.
   Count the tests with `wc -l`; report them with `grep`.
   Thus, it is of lesser importance to report such information
   by itself.

2. There are other numbers related to other reasons why a test might
   not be executed that may also be interesting. Of these, maybe
   the information from the 'requires' filter is the most interesting
   because it is the least obvious. and the least explicitly dependent 
   on command-line options.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/217#issuecomment-2292520849