RFR: 7903990: IDEA plugin: Itemize junit test results
Jorn Vernee
jvernee at openjdk.org
Thu Apr 17 12:22:04 UTC 2025
Itemize the results of jtreg tests containing junit sections. Each junit section will have a node in the test result tree, with each nested test class and test method having their own node as well. This also makes it possible to re-run just a single junit test (or a single iteration of a parameterized test), by right-clicking the corresponding item in the test result UI. See the below image as an example:

Some implementation notes: jtharness/jtreg don't currently report results of individual junit tests to the test listener. There are currently 2 barriers to this that I can see:
1. the JUnitRunner communicates the test results back to the parent jtreg process by writing a summary to stderr, which the parent process than parses. The parent process does not parse the results of the individual tests. In order to have more 'live feedback' from the junit runner, one idea is to use a socket connection between the parent jtreg process and junit runner. Where instead of writing the test results to stderr, the junit runner would call back to the parent process over this socket connection to notify it when a test started/finished. The jtreg parent process can then in turn immediately notify an test listener. Alternatively, we could perhaps use (existing) junit JFR events that are streamed back to the parent process.
2. jtreg does not report nested events of a test (such as junit tests) back to the harness observer that the plugin uses to listen for test events. In jtharness, a 'test' corresponds directly to a jtreg test, and there seems to be no support to attach additional nested events to a 'test', or for the JTRegTestListener to register a callback to listen for more nested events.
Rather than solving these two issues, for now I've opted for a simpler approach. When a jtreg test finishes and the test listener is notified, we look for sections in the test result titled 'junit', and parse their stderr stream to find the tests results that jtreg's JUnitRunner prints to stderr. This works relatively well, with the caveat being that, if jtharness truncates the stderr output, some tests that ran will not be reported in the UI in an itemized way (but the test will still fail, and report the failure under the 'jtreg' test, accompanied by its jtr file). Given how useful this feature seems (and has been during testing), I think this is a reasonable tradeoff.
-------------
Commit messages:
- bump copyright years
- polish
- add duration to junit tests
- improve code organization
- Use Section api
- polish
- make parsing more robust
- Make parsing jtr file safer
- Use custom test locator
- WIP - Itemized Results: sections
- ... and 1 more: https://git.openjdk.org/jtreg/compare/19cdb840...d92de904
Changes: https://git.openjdk.org/jtreg/pull/260/files
Webrev: https://webrevs.openjdk.org/?repo=jtreg&pr=260&range=00
Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903990
Stats: 396 lines in 4 files changed: 340 ins; 20 del; 36 mod
Patch: https://git.openjdk.org/jtreg/pull/260.diff
Fetch: git fetch https://git.openjdk.org/jtreg.git pull/260/head:pull/260
PR: https://git.openjdk.org/jtreg/pull/260
More information about the jtreg-dev
mailing list