RFR: 7903324: Improve per-class reporting of JUnit tests, in .jtr file

[Christian Stein]

On Mon, 26 Sep 2022 10:07:18 GMT, Christian Stein <cstein at openjdk.org> wrote:

> Improve per-class reporting of JUnit tests by using a custom `TestExecutionListener` in `JUnitRunner`.

Here are some sample output snippets copied from `.jtr` files collected by running self-tests if jtreg.

_Regenerating..._

Here's an output sample for a modified [JUnit Trace self-test](https://github.com/openjdk/jtreg/blob/master/test/junitTrace/JupiterTests.java) with a failing case, namely using `"123"` as a non-blank data point:

SUCCESSFUL: JupiterTests::nullEmptyAndBlankStrings '[1] test('null')'
SUCCESSFUL: JupiterTests::nullEmptyAndBlankStrings '[2] test('')'
SUCCESSFUL: JupiterTests::nullEmptyAndBlankStrings '[3] test(' ')'
SUCCESSFUL: JupiterTests::nullEmptyAndBlankStrings '[4] test('  ')'
    FAILED: JupiterTests::nullEmptyAndBlankStrings '[5] test('123')'
org.opentest4j.AssertionFailedError: Input text "123" not blank ==> expected: <true> but was: <false>
	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
	at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:40)
	at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:210)
	at JupiterTests.nullEmptyAndBlankStrings(JupiterTests.java:63)
[...]
	at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
	at java.base/java.lang.Thread.run(Thread.java:833)
SUCCESSFUL: JupiterTests::nullEmptyAndBlankStrings '[6] test('	')'
SUCCESSFUL: JupiterTests::succeedingTest 'succeedingTest()'
   ABORTED: JupiterTests::abortedTest 'abortedTest()'
org.opentest4j.TestAbortedException: Assumption failed: abort test execution mid-flight
JavaTest Message: JUnit Platform Failure(s): 1

[ JUnit Containers: found 4, started 4, succeeded 4, failed 0, aborted 0, skipped 0]
[ JUnit Tests: found 9, started 8, succeeded 6, failed 1, aborted 1, skipped 1]

java.lang.Exception: JUnit test failure
	at com.sun.javatest.regtest.agent.JUnitRunner.runWithJUnitPlatform(JUnitRunner.java:154)
	at com.sun.javatest.regtest.agent.JUnitRunner.main(JUnitRunner.java:96)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
	at java.base/java.lang.Thread.run(Thread.java:833)

JavaTest Message: Test threw exception: java.lang.Exception
JavaTest Message: shutting down test


Note: I pruned a lot of lines from the exception stack trace.

src/share/classes/com/sun/javatest/regtest/agent/JUnitRunner.java line 205:

> 203:                     // additionally print stacktrace for a non-successful test
> 204:                     if (result.getStatus() != TestExecutionResult.Status.SUCCESSFUL) {
> 205:                         result.getThrowable().ifPresent(printer::println);

Print complete stacktrace of the cause? Prune "Java system" frames?

src/share/classes/com/sun/javatest/regtest/agent/JUnitRunner.java line 228:

> 226: 
> 227:         @Override
> 228:         public void reportingEntryPublished(TestIdentifier identifier, ReportEntry entry) {

@AlanBateman Here is the place where a report entry is printed in the context of a test method.

-------------

PR: https://git.openjdk.org/jtreg/pull/127