[9] RFR: 8074580: sun/security/pkcs11/rsa/TestKeyPairGenerator.java fails due to PKCS11Exception: CKR_FUNCTION_FAILED

[Artem Smotrakov]

Hello,

Please review the following patch below for 9.

TestKeyPairGenerator.java test intermittently fails with 
CKR_FUNCTION_FAILED error when NSS crypto libs is used via SunPKCS11 
provider. Looks like the root cause is a bug 1012786 in NSS

https://bugzilla.mozilla.org/show_bug.cgi?id=1012786

RSA_NewKey() funсtion may fail with SEC_ERROR_NEED_RANDOM in case of 
lack of entropy which results to CKR_FUNCTION_FAILED.

PKCS11 spec says the following about CKR_FUNCTION_FAILED error:

ftp://ftp.rsasecurity.com/pub/pkcs/pkcs-11/v2-30/pkcs-11v2-30b-d6.pdf

...
CKR_FUNCTION_FAILED: ... In any event, although the function call
failed, the situation is not necessarily totally hopeless, as it is 
likely to be when
CKR_GENERAL_ERROR is returned. Depending on what the root cause of the
error actually was, it is possible that an attempt to make the exact 
same function call
again would succeed.
...

In this particular case, CKR_FUNCTION_FAILED error doesn't seem to be 
hopeless, and next call may succeed. The patch below tries to workaround 
this problem by calling C_GenerateKeyPair() function again in case of 
CKR_GENERAL_ERROR.

I was not able to reproduce this failure manually on several machines, 
so unfortunately I am not completely sure it solves the problem. I can 
back out the fix if we see those failures again.

Webrev: http://cr.openjdk.java.net/~asmotrak/8074580/webrev.00/
Bug: https://bugs.openjdk.java.net/browse/JDK-8074580

Artem