[9] RFR: JDK-8164322: sun/security/pkcs11/PKCS11Test.java shall be updated to run on ARM platforms

[Denis Kononenko]

Hi All,

> On 9/28/2016 2:02 AM, Artem Smotrakov wrote:
> > Currently the tests silently quit which looks like they pass. This
> makes
> > people think that everything went smoothly, but actually nothing
> was
> > tested.
> >
> I did not get the idea.  Looks like, if no NSS installed, the test
> would 
> be ignored; if NSS get installed, the test is actually get run.  If no
> 
> NSS get installed,  the test should quite silently and test nothing 
> because nothing should be tested.  That's the expected common behavior
> 
> in order to test specific configuration.

The problem is the tests report they passed but actually they were skipped. I have no objections against skipping tests but it would be better to give a hint somehow how many tests were skipped and why. 

> 
> If you don't want the test quit silently, I may prefer to check the 
> platform to install NSS libs rather than update this test cases. 
> Exposing the testing environment configuration problem is not the job
> of 
> this test.

There're 60+ tests related to PKCS11. Two years they have been "passed" on 3 unsupported platforms on hosts where usually no NSS libraries were installed. How can we rely on these results?

> 
> > I would prefer to update PKCS11Test to report a failure in case of
> > unexpected platform.
> Then you need to know all expected platform.  The test is not only run
> 
> in a certain known environment (for example the platforms Mach5 or 
> JPRT), it can also be run by third party environment (OpenJDK 
> contributors).  If not possible, I think it is a hard job to know all
> 
> the expected platform exactly.

>From my experience the most common environment issue is a lack of GUI libraries. Fortunately, in such case tests simply fail due to unsatisfied dependencies. If they would pass due to uncertain environment we could end up having completely untested UI functionality with green status.

> 
> Xuelei

Thank you,
Denis.