[9] RFR: 8074580: sun/security/pkcs11/rsa/TestKeyPairGenerator.java fails due to PKCS11Exception: CKR_FUNCTION_FAILED

[Artem Smotrakov]

I forgot to include PKCS11.java to webrev, here is an updated webrev

http://cr.openjdk.java.net/~asmotrak/8074580/webrev.01/

Artem

On 06/20/2016 11:12 AM, Artem Smotrakov wrote:
> Hello,
>
> Please review the following patch below for 9.
>
> TestKeyPairGenerator.java test intermittently fails with 
> CKR_FUNCTION_FAILED error when NSS crypto libs is used via SunPKCS11 
> provider. Looks like the root cause is a bug 1012786 in NSS
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=1012786
>
> RSA_NewKey() funсtion may fail with SEC_ERROR_NEED_RANDOM in case of 
> lack of entropy which results to CKR_FUNCTION_FAILED.
>
> PKCS11 spec says the following about CKR_FUNCTION_FAILED error:
>
> ftp://ftp.rsasecurity.com/pub/pkcs/pkcs-11/v2-30/pkcs-11v2-30b-d6.pdf
>
> ...
> CKR_FUNCTION_FAILED: ... In any event, although the function call
> failed, the situation is not necessarily totally hopeless, as it is 
> likely to be when
> CKR_GENERAL_ERROR is returned. Depending on what the root cause of the
> error actually was, it is possible that an attempt to make the exact 
> same function call
> again would succeed.
> ...
>
> In this particular case, CKR_FUNCTION_FAILED error doesn't seem to be 
> hopeless, and next call may succeed. The patch below tries to 
> workaround this problem by calling C_GenerateKeyPair() function again 
> in case of CKR_GENERAL_ERROR.
>
> I was not able to reproduce this failure manually on several machines, 
> so unfortunately I am not completely sure it solves the problem. I can 
> back out the fix if we see those failures again.
>
> Webrev: http://cr.openjdk.java.net/~asmotrak/8074580/webrev.00/
> Bug: https://bugs.openjdk.java.net/browse/JDK-8074580
>
> Artem