[9] RFR: JDK-8164322: sun/security/pkcs11/PKCS11Test.java shall be updated to run on ARM platforms
Artem Smotrakov
artem.smotrakov at oracle.com
Wed Sep 28 20:32:26 UTC 2016
Hi Xuelei,
I understand your concerns. But I'd prefer to be aware of situations
when a test reports that it passed when it actually did nothing.
How about using @requires key? We can try to specify all expected
platforms. If I understand correctly, jtreg won't run tests if specified
requirements are not met. In this case, you need to look at the report
to make sure all tests you are interested in were run.
Artem
On 09/28/2016 08:00 AM, Xuelei Fan wrote:
> On 9/28/2016 9:20 PM, Denis Kononenko wrote:
>> There're 60+ tests related to PKCS11. Two years they have been
>> "passed" on 3 unsupported platforms on hosts where usually no NSS
>> libraries were installed. How can we rely on these results?
> ;-) The words of "unsupported platforms" are very confusing in this
> mail thread.
>
> Let's think more about what if a test failed. What one will do if a
> test failed?
> 1. Test fail means source code problems for developers. One cannot
> submit a change-set if a test failed. He need to pay additional
> effort and analysis the failure. It take one developer a lot effort
> to know the root cause. I would never like this unnecessary cost.
> 2. In order to get the test pass, he need to install the NSS libs
> although NSS has nothing to do with his changeset. It may be a very
> very hard step or even impossible (for example licenses issues) step
> for him.
>
> TBH, I did not see much benefits to fail on unsupported platforms. I
> agree that skip for pass is not a good idea, but fail to warn is worse.
>
> I think the root cause if "unsupported platforms" actually are
> supported platforms, but by accident the NSS libraries are not
> installed or not installed properly.
>
> If one is not interested in NSS, the test get ignored (passed). If one
> is interested in NSS, he should install the NSS libs and the test get
> checked.
>
> What do you think if fix the testing infrastructure with properly
> installed NSS libs?
>
> > The problem is the tests report they passed but actually they were
> > skipped. I have no objections against skipping tests but it would
> > be better to give a hint somehow how many tests were skipped and why.
> Agreed. Unfortunately, there are only two options, "pass" or "fail",
> at present. It would be nice if there is a grey area. Any idea to
> make summary of skipped tests and reasons?
>
> Xuelei
More information about the security-dev
mailing list