RFR: CODETOOLS-7903267: Support executing a single test method in a JUnit class

[Jonathan Gibbons]

On Sat, 10 Sep 2022 01:13:21 GMT, Iris Clark <iris at openjdk.org> wrote:

>> Please review a medium-sized patch to support new feature that provides a way to specify that only parts of a test should be executed.  For JUnit tests, this allows a single method in the test to be executed, but the underlying mechanism is more general than that and can be leveraged by any framework that supports running many test cases within an overall test.
>> 
>> The feature is enabled by using a new syntactic form on the command line, informally called _query syntax_: `path-to-test?string`. The syntax is based on URL syntax, but note that unlike URLs, the query component comes at the end, and not before any fragment component. In other words, the full syntax is `path#id?string`. Generally, this reflects the hierarchical structure:   a path to a file, a test description within the file, and a string to specify part of the test to be run.
>> 
>> In terms of implementation, the "heavy lifting" is done in `TestManager`, to track the _file, id, query_ tuple, as compared to the previous _file, id_.  New abstractions `TestSpec` and `GroupSpec` are introduced, to replace stylistic use of simple strings.
>> 
>> The underlying JT Harness does not know (or need to know) about query components and so is unchanged. But the back end `RegressionScript` used to execute each test _does_ need to know the query component, and so the information about which tests involve a query component is tunneled through the `RegressionParameters` in a side channel, embodied by a new resource in the object. In itself, the `RegressionScript` has little to do with the query component, it is simply passed down to the actions of the test in a new `test.query` property or `TESTQUERY` environment variable.
>> 
>> `JUnitRunner` is updated to check for the new `test.query` property: if set, it is used as the name of a test method to be run. Note: a JUnit exception will be thrown and the test will fail if the method is not found. The exception does contain enough useful information to diagnose the issue, but we may want to detect and report the issue in a more friendly form in a followup change.
>> 
>> Finally, new jtreg self-tests are added to exercise the new feature.
>
> test/junitQueryTest/JUnitQueryTest.gmk line 233:
> 
>> 231: 
>> 232: TESTS.jtreg += $(BUILDTESTDIR)/JUnitQueryTest.invalidQuery.ok
>> 233: 
> 
> I'm sure there are others, but behavior when the query is associated with a valid test springs to mind, i.e.
> 
> Negative test: a/b/c/Test1.java?m14 
> 
> Also test: a/b/c/Test1.java?m14 and a/b/c/Test.java

I can add a negative test, such as for `?m14` but you just get an exception from JUnit about "method not found", so it counts as a "vanilla" test failure.

This would extend to the second proposed example: the tests in the file without a query would be executed, and the test with a bad query would fail.

Thinking about whether `jtreg` could detect and report an *error* instead of the test *failed*, I guess we _could_ check for the existence of the method in `JUnitAction` (i.e. within jtreg itself) but this would require that `jtreg` is both compiled and run with a JDK at least as new as the class being tested ... and that would be bad.

Once we are in the test runner or test code, the only options are to pass or fail -- i.e. return normally or throw an exception. We could carve out a special case similar to `jtreg.SkipException` but I'm not convinced it is worth the effort.   If the user gives a bad name, you get an exception.


TEST RESULT: Failed. Execution failed: `main' threw exception: org.junit.platform.commons.JUnitException: TestEngine with ID 'junit-jupiter' failed to discover tests



org.junit.platform.commons.JUnitException: TestEngine with ID 'junit-jupiter' failed to discover tests
Caused by: org.junit.platform.commons.JUnitException: MethodSelector [className = 'Test1', methodName = 'm14', methodParameterTypes = ''] resolution failed
	... 15 more
Caused by: org.junit.platform.commons.PreconditionViolationException: Could not find method with name [m14] in class [Test1].
	... 20 more

-------------

PR: https://git.openjdk.org/jtreg/pull/115