Certification of Preview Features

[Brian Goetz]

> With the latest Java SE 12 TCK, this testing looks as follows (see
> attached image):

Image was stripped.

> First of all, "may need" is a much to weak requirement. According to
> JEP 12 and our discussions around the topic I'm pretty sure that a
> Java SE 12 compliant implementation will "HAVE TO" run the tests in
> both modes.

Yes, this seems more in the spirit of preview features.  More precisely, some tests _must_ run with preview features enabled (in order to test the preview features) and some must _not_ run with the preview features (in order to test that they are not accidentally supported) — and most tests should pass in both.  

> The only practical solution to this problem would be to change the TCK
> such that it automatically runs all the required tests affected by the
> preview features in both configurations. Running all available tests
> in both configurations makes no sense and is a complete waste of time
> (e.g. why should one have to run all the interactive AWT/Swing or
> printing tests two times, with and without preview features enabled if
> they are not affected at all by these features?).

I disagree; a compliant implementation should pass the tests that do not rely on preview features in both modes.  What if an implementation, in preview mode, implemented completely different semantics for a standard feature?  If you only had to pass the “preview” tests in preview mode, such an implementation would appear compliant, even though this is grossly inconsistent with the spirit.  

> Finally, I do realize now that my proposal to use preview features at
> least within the reference implementation itself, in order to gather
> some experience with the new features, doesn't play very well with
> certification. Also, if the JDK itself would make use of such
> features, it would only run with "--enable-preview" which is probably
> not what we want (unless we find a way to enable preview features
> transparently for the JDK classes only).

Right.  That’s why we don’t do this.  Its a shame that we lose one of our best channels for feedback here, but that’s how the cookie bounces.