RFR: 8325247: Memory leak in SessionKeyRef class def when using PKCS11 security provider [v2]

[Valerie Peng]

On Mon, 19 Feb 2024 10:23:06 GMT, Daniel Jeliński <djelinski at openjdk.org> wrote:

>> The reported leak was caused by the death of the `Cleanup-SunPKCS11` thread. The cleanup thread in turn died because of an exception thrown from `removeNativeKey` that resulted from 2 threads executing that method at the same time.
>> 
>> This PR adds a reachabilityFence to ensure that the key will only be enqueued for cleanup after the user thread is done with the `removeNativeKey` call.
>> 
>> No new regression test; the issue is extremely hard to reproduce in a reasonable time. Existing tier1-3 tests continue to pass.
>> 
>> In JBS I attached a PoC patch that changes the relative timing of operations; with that patch and without the changes from this PR I am able to reproduce the issue within a few seconds. With the changes from this PR the issue did not reproduce after 10 minutes of testing.
>
> Daniel Jeliński has updated the pull request incrementally with one additional commit since the last revision:
> 
>   put reachabilityFence in a finally clause

Looks good, thanks for the fix~

-------------

Marked as reviewed by valeriep (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/17870#pullrequestreview-1894784874