Certification of Preview Features

[Volker Simonis]

Just for reference I've uploaded the stripped picture here:

http://cr.openjdk.java.net/~simonis/webrevs/2019/certi_preview.jpg

On Tue, Mar 19, 2019 at 11:12 AM Andrew Haley <aph at redhat.com> wrote:
>
> On 3/18/19 5:01 PM, Brian Goetz wrote:
>
> > I disagree; a compliant implementation should pass the tests that do
> > not rely on preview features in both modes.  What if an
> > implementation, in preview mode, implemented completely different
> > semantics for a standard feature?  If you only had to pass the
> > “preview” tests in preview mode, such an implementation would appear
> > compliant, even though this is grossly inconsistent with the spirit.
>
> Should pass, yes. But I read Volker's question in the spirit of
> practicality: we cannot test the JDK in a reasonable time in all
> possible modes.
>

Yes, it is mostly about usability/practicality. Already now it takes
an experienced engineer who is mostly doing certifications about two
full days to do a complete, compliant certification. Now multiply this
with the number of platforms and Java versions and you'll get a
considerable effort. The "Preview Feature" testing in its current form
will nearly double this effort one more time.

The other thing is that already now it is quite complex to do a
"compliant certification". And because nobody is allowed to share his
certification results, it is impossible to judge if somebody who
claims he has certified Java SE, really did a "compliant
certification". We already have too many knobs and settings which have
to be manually set in the correct way anyway. The new settings and
instructions for the Preview features (as shown in the picture)
further complicate certifications.

What I was suggesting is that the JCK test harness should run a
"reasonable amount" of tests in both modes automatically (without any
interaction from the tester). "Reasonable amount" is a test set
determined by the authors of the Preview Feature tests suite. Putting
a little more effort into the creation of the TCK will save huge
amounts of effort during certification and make the certification
results more reliable.

As a prominent example, I really see absolutely no reason in running
the interactive AWT/Swing test (which require several hours of
tedious, manual work) two times just because of the "Switch
Expressions" preview feature.

> --
> Andrew Haley
> Java Platform Lead Engineer
> Red Hat UK Ltd. <https://www.redhat.com>
> EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671