RFR: 8324585: JVM native memory leak in PCKS11-NSS security provider

[Valerie Peng]

On Fri, 26 Jan 2024 10:04:11 GMT, Daniel Jeliński <djelinski at openjdk.org> wrote:

> Please review this patch that fixes a memory leak in P11TlsPrfGenerator, which is triggered during TLS1.2 Finished message generation and verification.
> 
> The patch changes C_SignInit JNI method to free the mechanism data immediately after use. This matches the behavior of other Init methods (like C_EncryptInit). The patch also fixes a similar issue in other signature-related methods.
> 
> The change essentially reverts part of [JDK-8080462](https://bugs.openjdk.org/browse/JDK-8080462).
> 
> All sun/security/pkcs11 tests still pass with NSS 3.35 and 3.91. All tier1-3 tests still pass.

Marked as reviewed by valeriep (Reviewer).

IIRC, this may be the special handling to work around the PSS errors I observed when implementing the support. Good that we don't need them now.

-------------

PR Review: https://git.openjdk.org/jdk/pull/17584#pullrequestreview-1846619636
PR Comment: https://git.openjdk.org/jdk/pull/17584#issuecomment-1912757799