testing the -target option

[Joseph D. Darcy]

On 10/31/2024 1:29 PM, Alan Snyder wrote:
>>
>> You can satisfy your curiosity by looking the the javac regressions 
>> tests in the OpenJDK repo. Under
>>
>> https://github.com/openjdk/jdk
>>
>> see the test/langtools/tools/javac directory hierarchy.
>>
> Thank you.
>
> I took a quick look at the files in that directory. I found many 
> examples of using -target (or --release). The negative tests are easy 
> to understand. The feature tests less so, as apparently each 
> individual test developer decides what releases to test.


When I write tests for a new language feature, I'll use multiple test 
harness directives in a single test file:

1) Run the test on the last version of the language _without_ the 
feature and verify the right compiler error is reported at the expected 
location
2) Run the test on the current language version, which is assumed to 
support the feature going into the future, and in some cases verify the 
operation of compiled code

Correct functional operation of the feature, i.e. does the generated 
code run properly, may also be done under separate test files.

While not implemented this way directly, the "is feature X supported in 
the language level being used for this compilation" usually boils down 
to a check of "language level used for the compilation >= first language 
level where this feature is supported."

>
> The thoroughness of the tests varies considerably. A few tests use all 
> of the supported releases (from 8 onward), but many just test one or 
> two releases and the current release. I don't see any way to estimate 
> the coverage of these tests.

For a given feature, generally it is either support or not supported 
and, if it is supported, it is most commonly compiled a single way 
across releases (although as noted elsewhere, that is not a 
requirement). So as white box regression tests, this technique is 
sufficient for code coverage.


>
> Given that the byte code produced by javac is not examined, the best 
> way to positively test the -target feature for a given target is to 
> run tests that are compatible with that target, and I would think that 
> the best source of such tests is the actual target release. It looks 
> like jtreg could do this, using the -compilejdk and -javacoption 
> options. Has anyone tried this?

A considerable amount of effort is already spent on testing the JDK, 
especially "tier 1" areas like the core libraries and javac. Spinning up 
large new testing methodologies that have little expectation of finding 
issues is not attractive.

-Joe