RFR: 7903990: IDEA plugin: Itemize junit test results

[Jorn Vernee]

On Tue, 15 Apr 2025 20:48:35 GMT, Jorn Vernee <jvernee at openjdk.org> wrote:

> Itemize the results of jtreg tests containing junit sections. Each junit section will have a node in the test result tree, with each nested test class and test method having their own node as well. This also makes it possible to re-run just a single junit test (or a single iteration of a parameterized test), by right-clicking the corresponding item in the test result UI. See the below image as an example:
> 
> ![image](https://github.com/user-attachments/assets/649b5899-47dd-451e-acda-7d5e21824f4d)
> 
> Some implementation notes: jtharness/jtreg don't currently report results of individual junit tests to the test listener. There are currently 2 barriers to this that I can see: 
> 
> 1. the JUnitRunner communicates the test results back to the parent jtreg process by writing a summary to stderr, which the parent process than parses. The parent process does not parse the results of the individual tests. In order to have more 'live feedback' from the junit runner, one idea is to use a socket connection between the parent jtreg process and junit runner. Where instead of writing the test results to stderr, the junit runner would call back to the parent process over this socket connection to notify it when a test started/finished. The jtreg parent process can then in turn immediately notify an test listener. Alternatively, we could perhaps use (existing) junit JFR events that are streamed back to the parent process. 
> 2. jtreg does not report nested events of a test (such as junit tests) back to the harness observer that the plugin uses to listen for test events. In jtharness, a 'test' corresponds directly to a jtreg test, and there seems to be no support to attach additional nested events to a 'test', or for the JTRegTestListener to register a callback to listen for more nested events.
> 
> Rather than solving these two issues, for now I've opted for a simpler approach. When a jtreg test finishes and the test listener is notified, we look for sections in the test result titled 'junit', and parse their stderr stream to find the tests results that jtreg's JUnitRunner prints to stderr. This works relatively well, with the caveat being that, if jtharness truncates the stderr output, some tests that ran will not be reported in the UI in an itemized way (but the test will still fail, and report the failure under the 'jtreg' test, accompanied by its jtr file). Given how useful this feature seems (and has been during testing), I think this is a reasonable tradeoff.

Another caveat of this feature I've run into, or rather, what seems to be a limitation imposed by IntelliJ: in order to show the right-click dialogue to re-run a test case, we need to look up the location of the test class/method, which then gets passed back to the plugin later to create the run configuration. This involves at least looking up a class using IntelliJ's builtin lookup functionality.

The issue is: when we look up the name of the test class, the file it is in needs to be in an active IntelliJ test source root. The plugin will normally automatically set up the test source root when opening a test file, but if you forget to do this before running the test, the test class can not be found, and instead of the dialogue from the screen shot, a dialogue with the text 'Nothing here' will be shown instead.

![image](https://github.com/user-attachments/assets/5f3afc37-fcb6-4d3b-9776-4eb4e86e2c51)

The worse thing is that this lookup result seems to be cached inside the IDE, so even if you add the test source root after the seeing this, the IDE will keep showing the 'Nothing here' dialogue. The only way I've found to resolve that issue is to restart the IDE.

In other words: it is important to make sure the test you're running are in an active test source root _before_ running the tests.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/260#issuecomment-2814083451