RFR: 7903990: IDEA plugin: Itemize junit test results

[Jorn Vernee]

On Tue, 15 Apr 2025 20:48:35 GMT, Jorn Vernee <jvernee at openjdk.org> wrote:

> Itemize the results of jtreg tests containing junit sections. Each junit section will have a node in the test result tree, with each nested test class and test method having their own node as well. This also makes it possible to re-run just a single junit test (or a single iteration of a parameterized test), by right-clicking the corresponding item in the test result UI. See the below image as an example:
> 
> ![image](https://github.com/user-attachments/assets/649b5899-47dd-451e-acda-7d5e21824f4d)
> 
> Some implementation notes: jtharness/jtreg don't currently report results of individual junit tests to the test listener. There are currently 2 barriers to this that I can see: 
> 
> 1. the JUnitRunner communicates the test results back to the parent jtreg process by writing a summary to stderr, which the parent process than parses. The parent process does not parse the results of the individual tests. In order to have more 'live feedback' from the junit runner, one idea is to use a socket connection between the parent jtreg process and junit runner. Where instead of writing the test results to stderr, the junit runner would call back to the parent process over this socket connection to notify it when a test started/finished. The jtreg parent process can then in turn immediately notify an test listener. Alternatively, we could perhaps use (existing) junit JFR events that are streamed back to the parent process. 
> 2. jtreg does not report nested events of a test (such as junit tests) back to the harness observer that the plugin uses to listen for test events. In jtharness, a 'test' corresponds directly to a jtreg test, and there seems to be no support to attach additional nested events to a 'test', or for the JTRegTestListener to register a callback to listen for more nested events.
> 
> Rather than solving these two issues, for now I've opted for a simpler approach. When a jtreg test finishes and the test listener is notified, we look for sections in the test result titled 'junit', and parse their stderr stream to find the tests results that jtreg's JUnitRunner prints to stderr. This works relatively well, with the caveat being that, if jtharness truncates the stderr output, some tests that ran will not be reported in the UI in an itemized way (but the test will still fail, and report the failure under the 'jtreg' test, accompanied by its jtr file). Given how useful this feature seems (and has been during testing), I think this is a reasonable tradeoff.

There's one more issue I've discovered: when running tests concurrently (using e.g. `-conc:auto`), the tests will get nested into each other:

![image](https://github.com/user-attachments/assets/fff2b995-250d-4dd5-ad40-275c5b3ceba5)

I've made several attempts to work around this issue, for instance by assigning a different 'flow id' to each test (as described [here](https://www.jetbrains.com/help/teamcity/service-messages.html#Message+FlowId)), but this parameter seems to be ignored by intellij (which I've found out by debugging the IDE code itself) (see e.g. [here](https://github.com/JetBrains/intellij-community/blob/40473f1b030aa26244a8051df8d8a90fdbafbec6/platform/smRunner/src/com/intellij/execution/testframework/sm/runner/GeneralToSMTRunnerEventsConvertor.java#L139) where the code only has a single flowId-insensitive stack of test suites.) It seems to be possible to create the entire event tree up front, and then the actual test messages just attach to the already existing tree. This is what the junit runner does, but unlike the junit runner, we don't know the structure of the event tree before we start running tests, so this isn't really an option for us.

I don't think concurrent test running ever really  worked in the plugin. With the old code we don't get the weird nesting, but each new test started would immediately end an already running test. For now I think the best solution to make concurrent tests work with the new setup, is to only report the 'test suite started' message from `finishedTest`, under a lock. This gets us a correct event tree, with the downside that tests only start appearing in the UI once they are finished. Seems like a decent tradeoff?

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/260#issuecomment-2836964176