Certification of Preview Features

[Volker Simonis]

Hi,

we've recently started the TCK certification for Java SE 12. As we all
know, Java SE 12 is the first release with "Preview Features". As
we've previously discussed and as described in the "JEP 12: Preview
Language and VM Features" [1], preview features are fully specified
and fully implemented features of the Java SE platform which have to
be certified by the corresponding TCK.

With the latest Java SE 12 TCK, this testing looks as follows (see
attached image):

- one has to choose whether to run the test on the preview features or not
- the documentation mentions that one "may need to test the product in
both modes"

First of all, "may need" is a much to weak requirement. According to
JEP 12 and our discussions around the topic I'm pretty sure that a
Java SE 12 compliant implementation will "HAVE TO" run the tests in
both modes.

Second, the TCK doesn't mention which tests will have to be run in
"both modes". Running all the runtime and langtools tests (they have
the same "preview feature" switch) is a real big overhead. It actually
doubles the time for a correct certification (which already is quite
substantial). It also interferes with other test switches like
"verify/noverify" which already require to run a part of the tests in
two different configurations. Together with the preview features, we
will now theoretically get four configurations which will have to be
tested.

The only practical solution to this problem would be to change the TCK
such that it automatically runs all the required tests affected by the
preview features in both configurations. Running all available tests
in both configurations makes no sense and is a complete waste of time
(e.g. why should one have to run all the interactive AWT/Swing or
printing tests two times, with and without preview features enabled if
they are not affected at all by these features?).

Finally, I do realize now that my proposal to use preview features at
least within the reference implementation itself, in order to gather
some experience with the new features, doesn't play very well with
certification. Also, if the JDK itself would make use of such
features, it would only run with "--enable-preview" which is probably
not what we want (unless we find a way to enable preview features
transparently for the JDK classes only).

What do you think?

Regards,
Volker

[1] http://openjdk.java.net/jeps/12