[9] RFR: JDK-8164322: sun/security/pkcs11/PKCS11Test.java shall be updated to run on ARM platforms

[Xuelei Fan]

On 9/28/2016 9:20 PM, Denis Kononenko wrote:
> There're 60+ tests related to PKCS11. Two years they have been "passed" on 3 unsupported platforms on hosts where usually no NSS libraries were installed. How can we rely on these results?
;-) The words of "unsupported platforms" are very confusing in this mail 
thread.

Let's think more about what if a test failed.  What one will do if a 
test failed?
1. Test fail means source code problems for developers.  One cannot 
submit a change-set if a test failed.  He need to pay additional effort 
and analysis the failure.  It take one developer a lot effort to know 
the root cause.  I would never like this unnecessary cost.
2. In order to get the test pass, he need to install the NSS libs 
although NSS has nothing to do with his changeset.  It may be a very 
very hard step or even impossible (for example licenses issues) step for 
him.

TBH, I did not see much benefits to fail on unsupported platforms.  I 
agree that skip for pass is not a good idea, but fail to warn is worse.

I think the root cause if "unsupported platforms" actually are supported 
platforms, but by accident the NSS libraries are not installed or not 
installed properly.

If one is not interested in NSS, the test get ignored (passed).  If one 
is interested in NSS, he should install the NSS libs and the test get 
checked.

What do you think if fix the testing infrastructure with properly 
installed NSS libs?

 > The problem is the tests report they passed but actually they were
 > skipped. I have no objections against skipping tests but it would
 > be better to give a hint somehow how many tests were skipped and why.
Agreed.  Unfortunately, there are only two options, "pass" or "fail", at 
present.  It would be nice if there is a grey area.  Any idea to make 
summary of skipped tests and reasons?

Xuelei