Certification of Preview Features

Tim Ellison Tim_Ellison at uk.ibm.com
Tue Mar 19 21:46:57 UTC 2019


"Volker Simonis" <volker.simonis at gmail.com> wrote on 19/03/2019 19:48:46:

> Just for reference I've uploaded the stripped picture here:
> 
> https://urldefense.proofpoint.com/v2/url?
> 
u=http-3A__cr.openjdk.java.net_-7Esimonis_webrevs_2019_certi-5Fpreview.jpg&d=DwIFaQ&c=jf_iaSHvJObTbx-
> 
siA1ZOg&r=AtEh0PzjNUxLZESO8_yZ6DCI4DYOhKAFQsZXG3kYuYM&m=43zuj3fHvHrNSVl98AXrNTp2Yjh5Tenxf8mOOzzGKCY&s=i55Xf3ndRvtWkBbQPPvIa_fUY40THkVVz55WG6Fv-4M&e=
> 
> On Tue, Mar 19, 2019 at 11:12 AM Andrew Haley <aph at redhat.com> wrote:
> >
> > On 3/18/19 5:01 PM, Brian Goetz wrote:
> >
> > > I disagree; a compliant implementation should pass the tests that do
> > > not rely on preview features in both modes.  What if an
> > > implementation, in preview mode, implemented completely different
> > > semantics for a standard feature?  If you only had to pass the
> > > “preview” tests in preview mode, such an implementation would appear
> > > compliant, even though this is grossly inconsistent with the spirit.
> >
> > Should pass, yes. But I read Volker's question in the spirit of
> > practicality: we cannot test the JDK in a reasonable time in all
> > possible modes.
> >
> 
> Yes, it is mostly about usability/practicality. Already now it takes
> an experienced engineer who is mostly doing certifications about two
> full days to do a complete, compliant certification. Now multiply this
> with the number of platforms and Java versions and you'll get a
> considerable effort. The "Preview Feature" testing in its current form
> will nearly double this effort one more time.
> 
> The other thing is that already now it is quite complex to do a
> "compliant certification". And because nobody is allowed to share his
> certification results, it is impossible to judge if somebody who
> claims he has certified Java SE, really did a "compliant
> certification". We already have too many knobs and settings which have
> to be manually set in the correct way anyway. The new settings and
> instructions for the Preview features (as shown in the picture)
> further complicate certifications.
> 
> What I was suggesting is that the JCK test harness should run a
> "reasonable amount" of tests in both modes automatically (without any
> interaction from the tester). "Reasonable amount" is a test set
> determined by the authors of the Preview Feature tests suite. Putting
> a little more effort into the creation of the TCK will save huge
> amounts of effort during certification and make the certification
> results more reliable.

So there are (at least) two ways of looking at this:

The JCK authors already make decisions about how extensively they are
testing to the specification.  Of course, compliance testing cannot be
(mathematically) complete so the the JCK authors are using their skill
and judgement to decide the number and scope of tests for particular APIs
and features to determine spec compliance.

It sounds like you are arguing that the same informed opinion of the JCK
authors be used to decide the extent of testing under the Preview 
Features. 

However, where the JCK framework *can* be used to test exhaustively, such
as via setting modes, it makes sense to allow the tests to be run that way
without artificially limiting the tests that can be run in such modes. 
That
is part of the spec compliance definition.

It then becomes part of the specification compliance to require all
tests pass in all modes, but for the JCK users to decide where they are
confident in making the assertion of a pass.  If somebody subsequently
demonstrates that the user's confidence was misplaced, i.e. by 
demonstrating
a failure mode, then that's a bug to be fixed.

I suppose the analogy is: we declare releases complete when tested amongst
a representative set of dependent libraries in the OS, and declare 
confidence that the release will work in other combinations of OS library
versions etc without explicitly testing them.  If somebody shows that is
untrue we fix it.

> As a prominent example, I really see absolutely no reason in running
> the interactive AWT/Swing test (which require several hours of
> tedious, manual work) two times just because of the "Switch
> Expressions" preview feature.

You might make the judgement call not to run those tests and still declare
compliance; but if somebody can demonstrate that Switch Expressions 
actually
does break the interactive tests then I'm sure you would decide to fix it.

Regards,
Tim


Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU



More information about the java-se-spec-experts mailing list