Certification of Preview Features

[Brian Goetz]

I have some additional guidance here.  Hopefully this helps.  

The sole preview feature in Java SE 12 was switch expressions, which expand the Java language unconditionally. That means:

1. Of the programs that were legal in Java SE 11, all are legal in Java SE 12 whether preview features are disabled or enabled.

2. Of the programs that were illegal in Java SE 11, some are legal in Java SE 12 when preview features are enabled. (And some are still illegal in Java SE 12 whether preview features are disabled or enabled. For example, `switch (e) { case 1 ---> ...}` was illegal in Java SE 11 and is still illegal in Java SE 12 even with preview features enabled, because the arrow is written wrong.)

#1 means that JCK 12 did not invalidate or invert pre-existing tests for non-preview features. #2 means that JCK 12 added tests for preview features (some positive, some negative) that require preview features to be enabled. Since the two sets of tests are orthogonal, you don't have to run the whole test suite twice to certify conformance. In the Configuration Editor, you can:

1. Disable preview features, and test only non-preview features.

  Test Platform -> Run tests on Preview Features -> "No"
  Standard Values... -> Keywords -> `!switch_expressions`

2. Enable preview features, and test only preview features.

  Test Platform -> Run tests on Preview Features -> "Yes"
  Standard Values... -> Keywords -> `switch_expressions`

This incremental approach is already used for headless versus headful implementations. Every program that runs on a headless system will run on a headful system, and of the programs that won't run on a headless system, some will run on a headful system. Accordingly, the headless tests and the headful tests are orthogonal, and you don't have to run all of them twice to certify a headful implementation. The JCK harness lets you choose either the headless tests or the headless-plus-headful tests: in the Configuration Editor (Test Platform -> Headless System), you answer "No" to the interview question "Is the system you are testing headless?", and more tests will be executed than if you had answered "Yes".

Since preview features are mandatory, you might wonder why the JCK harness doesn't run all the tests and have preview features enabled all the time. The answer is "an abundance of caution" -- JCK tries to avoid assumptions about how preview features are implemented. The concern is that when preview features are enabled, an implementation might give the wrong result for a test of a NON-preview feature. This won't happen for the OpenJDK implementation of switch expressions, but it could happen for a third-party implementation of another preview feature in future. That's why JCK 12 left it up to the implementer to determine if the whole suite can run with preview features enabled.

In JCK 13, the User Guide will give more clarity on how to handle preview features' certification by using the keywords mentioned above. This should help avoid unnecessary runs when an implementation is a straightforward derivative of OpenJDK with no major changes to preview features' implementation.

Looking forward, a preview feature might not expand the Java language unconditionally. JEP 12 notes the case of the `_` character, whereby the space of Java identifiers decreased between Java SE 8 and 9. Some future JCK N would need four keywords:

(a) java_se_N_preview_language_features_positive_when_nopreview
(b) java_se_N_preview_language_features_negative_when_nopreview
(c) java_se_N_preview_language_features_positive_when_preview
(d) java_se_N_preview_language_features_negative_when_preview

In effect, JCK 12 added tests with the (c) and (d) keywords, both spelled `switch_expressions`. Support for "Drop _ as an identifier" would mean finding any pre-existing tests that exercise _ as an identifier, and adding the (a) or (b) keyword to each test as applicable; then copying all those tests and annotating the copies with the (d) keyword. (Longstanding JCK practice is that one test has one result, so if a test's result depends on the enablement of preview features, then what you need is two tests.)

There would be no interview question about preview features at all -- the JCK harness would run every compiler test with either no (a)-(d) keywords, or just (a) or (b), with preview features disabled; and run every compiler test with just (c) or (d) with preview features enabled.

> On Mar 18, 2019, at 11:59 AM, Volker Simonis <volker.simonis at gmail.com> wrote:
> 
> Hi,
> 
> we've recently started the TCK certification for Java SE 12. As we all
> know, Java SE 12 is the first release with "Preview Features". As
> we've previously discussed and as described in the "JEP 12: Preview
> Language and VM Features" [1], preview features are fully specified
> and fully implemented features of the Java SE platform which have to
> be certified by the corresponding TCK.
> 
> With the latest Java SE 12 TCK, this testing looks as follows (see
> attached image):
> 
> - one has to choose whether to run the test on the preview features or not
> - the documentation mentions that one "may need to test the product in
> both modes"
> 
> First of all, "may need" is a much to weak requirement. According to
> JEP 12 and our discussions around the topic I'm pretty sure that a
> Java SE 12 compliant implementation will "HAVE TO" run the tests in
> both modes.
> 
> Second, the TCK doesn't mention which tests will have to be run in
> "both modes". Running all the runtime and langtools tests (they have
> the same "preview feature" switch) is a real big overhead. It actually
> doubles the time for a correct certification (which already is quite
> substantial). It also interferes with other test switches like
> "verify/noverify" which already require to run a part of the tests in
> two different configurations. Together with the preview features, we
> will now theoretically get four configurations which will have to be
> tested.
> 
> The only practical solution to this problem would be to change the TCK
> such that it automatically runs all the required tests affected by the
> preview features in both configurations. Running all available tests
> in both configurations makes no sense and is a complete waste of time
> (e.g. why should one have to run all the interactive AWT/Swing or
> printing tests two times, with and without preview features enabled if
> they are not affected at all by these features?).
> 
> Finally, I do realize now that my proposal to use preview features at
> least within the reference implementation itself, in order to gather
> some experience with the new features, doesn't play very well with
> certification. Also, if the JDK itself would make use of such
> features, it would only run with "--enable-preview" which is probably
> not what we want (unless we find a way to enable preview features
> transparently for the JDK classes only).
> 
> What do you think?
> 
> Regards,
> Volker
> 
> [1] http://openjdk.java.net/jeps/12