RFR: 8324585: JVM native memory leak in PCKS11-NSS security provider

[Daniel Jeliński]

On Fri, 26 Jan 2024 10:04:11 GMT, Daniel Jeliński <djelinski at openjdk.org> wrote:

> Please review this patch that fixes a memory leak in P11TlsPrfGenerator, which is triggered during TLS1.2 Finished message generation and verification.
> 
> The patch changes C_SignInit JNI method to free the mechanism data immediately after use. This matches the behavior of other Init methods (like C_EncryptInit). The patch also fixes a similar issue in other signature-related methods.
> 
> The change essentially reverts part of [JDK-8080462](https://bugs.openjdk.org/browse/JDK-8080462).
> 
> All sun/security/pkcs11 tests still pass with NSS ~3.35 and~ 3.91. All tier1-3 tests still pass.
> 
> EDIT:
> Some sun/security/pkcs11 tests fail with NSS 3.64 and older, see [comment](https://github.com/openjdk/jdk/pull/17584#issuecomment-1914665234)

Updated the code to disable RSA-PSS on known bad NSS versions

-------------

PR Comment: https://git.openjdk.org/jdk/pull/17584#issuecomment-1918806644