[9] RFR: JDK-8164322: sun/security/pkcs11/PKCS11Test.java shall be updated to run on ARM platforms

[Artem Smotrakov]

Hi Xuelei,

This is not a problem with machine configuration. The actual test cases 
are not going to be run (even if there are NSS libs on a test machine) 
until "osMap" contains an entry for specific platform.

Artem


On 09/28/2016 05:52 PM, Xuelei Fan wrote:
> Hi Artem,
>
> What do you think fix the testing infrastructure (JPRT/Mach5) with 
> proper configuration?  I think it is a easier than update a large 
> bunch of test codes.
>
> I understand the concerns of yours, but I don't want add additional 
> effort for those who need to run the test on their own environment. 
> Some people run the test locally before submit JPRT jobs (or no access 
> to JPRT systems for external people).
>
> Update the testing infrastructure may solve both of your concerns and 
> my concerns.
>
> I'm not sure the @requires tag would work or not.  It would be great 
> if you can find a solution with the tag.
>
> Thanks,
> Xuelei
>
> On 9/29/2016 4:32 AM, Artem Smotrakov wrote:
>> Hi Xuelei,
>>
>> I understand your concerns. But I'd prefer to be aware of situations
>> when a test reports that it passed when it actually did nothing.
>>
>> How about using @requires key? We can try to specify all expected
>> platforms. If I understand correctly, jtreg won't run tests if specified
>> requirements are not met. In this case, you need to look at the report
>> to make sure all tests you are interested in were run.
>>
>> Artem
>>
>>
>> On 09/28/2016 08:00 AM, Xuelei Fan wrote:
>>> On 9/28/2016 9:20 PM, Denis Kononenko wrote:
>>>> There're 60+ tests related to PKCS11. Two years they have been
>>>> "passed" on 3 unsupported platforms on hosts where usually no NSS
>>>> libraries were installed. How can we rely on these results?
>>> ;-) The words of "unsupported platforms" are very confusing in this
>>> mail thread.
>>>
>>> Let's think more about what if a test failed.  What one will do if a
>>> test failed?
>>> 1. Test fail means source code problems for developers.  One cannot
>>> submit a change-set if a test failed.  He need to pay additional
>>> effort and analysis the failure.  It take one developer a lot effort
>>> to know the root cause.  I would never like this unnecessary cost.
>>> 2. In order to get the test pass, he need to install the NSS libs
>>> although NSS has nothing to do with his changeset.  It may be a very
>>> very hard step or even impossible (for example licenses issues) step
>>> for him.
>>>
>>> TBH, I did not see much benefits to fail on unsupported platforms.  I
>>> agree that skip for pass is not a good idea, but fail to warn is worse.
>>>
>>> I think the root cause if "unsupported platforms" actually are
>>> supported platforms, but by accident the NSS libraries are not
>>> installed or not installed properly.
>>>
>>> If one is not interested in NSS, the test get ignored (passed). If one
>>> is interested in NSS, he should install the NSS libs and the test get
>>> checked.
>>>
>>> What do you think if fix the testing infrastructure with properly
>>> installed NSS libs?
>>>
>>> > The problem is the tests report they passed but actually they were
>>> > skipped. I have no objections against skipping tests but it would
>>> > be better to give a hint somehow how many tests were skipped and why.
>>> Agreed.  Unfortunately, there are only two options, "pass" or "fail",
>>> at present.  It would be nice if there is a grey area. Any idea to
>>> make summary of skipped tests and reasons?
>>>
>>> Xuelei
>>