RFR: 7903785: Improve reporting for skipped tests
Jaikiran Pai
jpai at openjdk.org
Thu Aug 8 15:26:46 UTC 2024
On Wed, 7 Aug 2024 18:25:13 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:
> This is to support the ability to track `skipped` tests in the JDK `make test` system.
>
> It is somewhere between "proof of concept", "draft PR", and initial version of a real PR.
>
> The primary goal is to post the number of `skipped` tests in a backwards compatible way that does not break existing makefile rules or require any changes. The makefiles currently have some "fuzzy parsing" to search for named values that may appear on the `Test results:` line near the end of the `jtreg` output to the console. This PR _adds_ a new value to that line _without changing any existing values_. In particular, the `passed` value is unchanged and continues to include the `skipped` tests, as before, but for readers and tools alike, the number of skipped tests is also reported.
>
> The PR also exposes a previously undocumented feature, to control the formatting of that line, using a format string that can be given to `jtreg` in a system property. The set of format specifiers in that string has been extended to include format specifiers for "skipped tests" and "passed tests, excluding skipped tests".
>
> In exposing that feature, some design questions come to mind:
> * the terminology of "ignored" tests is somewhat confusing -- it is about tests that were not selected by virtue of the keyword filter (and related `-k` option)
> * if the number of `skipped` tests is interesting, there are other similar numbers that might also be interesting, such as the number of tests that were filtered out by a `@requires` tag -- which is similar to but more powerful than the keyword filter. But these extra numbers are contained within a composite filter, and may not have been so easily accessible when `jtreg` was built using older versions of JDK -- that is, before covariant returns.
> * the `conditional space` and `conditional comma` provided as format specifiers do not work as well as intended if there is a plain-text label at the beginning of the format string.
>
> But to summarize, this feature is primarily about reporting the number of `skipped` tests. We can either defer supporting additional values until a subsequent PR, or put more work into this PR to expose more of these "interesting" numbers.
Hello Jon, I haven't yet reviewed the code in detail, but have only looked at the PR description and the high level changes. With my limited knowledge of this project, the proposed idea looks good to me.
Does `excluded` cover only problem listed exclusions?
With the proposed change, I think we will be categorizing tests that haven't run the actual test logic into different categories - "excluded" (due to problem listing?), "skipped" (due to jtreg.SkippedException or test framework specific SkippedException type?), "ignored" (due to keyword evaluation). If later we do support "@requries" exclusion reporting, would that be a category of its own?
The other thing I think we should additionally consider is a summary report of these tests that haven't run. For example, if the count says 3 tests were skipped, 1 test was ignored, then I think we should have a notrun-summary.txt (keeping aside the file name for now) which would have something like:
skipped:
foo/bar/HelloTest.java threw jtreg.SkippedException
foo/bar/OneTwoThree.java threw testng.SkippedException
foo/bar/Five.java threw junit.AssumptionFailureException
ignored:
hello/world/HelloWorld.java due to keyword "intermittent"
-------------
PR Comment: https://git.openjdk.org/jtreg/pull/217#issuecomment-2276100333
More information about the jtreg-dev
mailing list