RFR: 8324585: JVM native memory leak in PCKS11-NSS security provider

[Daniel Jeliński]

On Fri, 26 Jan 2024 22:06:23 GMT, Valerie Peng <valeriep at openjdk.org> wrote:

>> Please review this patch that fixes a memory leak in P11TlsPrfGenerator, which is triggered during TLS1.2 Finished message generation and verification.
>> 
>> The patch changes C_SignInit JNI method to free the mechanism data immediately after use. This matches the behavior of other Init methods (like C_EncryptInit). The patch also fixes a similar issue in other signature-related methods.
>> 
>> The change essentially reverts part of [JDK-8080462](https://bugs.openjdk.org/browse/JDK-8080462).
>> 
>> All sun/security/pkcs11 tests still pass with NSS 3.35 and 3.91. All tier1-3 tests still pass.
>
> IIRC, this may be the special handling to work around the PSS errors I observed when implementing the support. Good that we don't need them now.

Thanks @valeriepeng for your review. I started looking into why I wasn't able to reproduce the errors you were seeing, and found that the tests I run with NSS 3.35 were silently skipped. I had to make some adjustments to PKCS11Test.java to actually make them work. I'll document that in a separate JBS ticket shortly.

Bottom line: With NSS 3.35 the following tests fail with this change:

sun/security/pkcs11/Signature/InitAgainPSS.java
sun/security/pkcs11/Signature/SigInteropPSS.java
sun/security/pkcs11/Signature/SignatureTestPSS.java
sun/security/pkcs11/sslecc/ClientJSSEServerJSSE.java

This was a NSS problem which was fixed here:
https://hg.mozilla.org/projects/nss/diff/be386bdafeb8dcfd894af7ff151b04afe748857a/lib/softoken/pkcs11c.c#l1.639
The fix was released in NSS 3.65.

Now, the still-supported Ubuntu 20.04 ships with NSS 3.49, which does not have this fix. I suppose other distros might also have non-EOL releases with a broken NSS version. How can we alert them about the problems they may face with this fix?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/17584#issuecomment-1914665234