Integrated: 8307185: pkcs11 native libraries make JNI calls into java code while holding GC lock

[Daniel Jeliński]

On Wed, 19 Jul 2023 12:03:14 GMT, Daniel Jeliński <djelinski at openjdk.org> wrote:

> This patch fixes random deadlocks in PKCS11 decryption and encryption code.
> 
> The deadlocks were caused by object allocation in `ckAssertReturnValueOK` waiting for GC; GC was in turn waiting for `ReleasePrimitiveArrayCritical`, which never happened.
> 
> The fix moves the call to `ckAssertReturnValueOK` after `ReleasePrimitiveArrayCritical`.
> 
> All tests in sun/security/pkcs11 were executed with `JTREG=JAVA_OPTIONS=-Xcheck:jni` with and without this patch.
> Without this patch, 5 tests produced the following warning:
> 
> Warning: Calling other JNI functions in the scope of Get/ReleasePrimitiveArrayCritical or Get/ReleaseStringCritical
> 
> With this patch the warning was not observed.
> 
> Tier2-3 tests are still green.

This pull request has now been integrated.

Changeset: 354c6605
Author:    Daniel Jeliński <djelinski at openjdk.org>
URL:       https://git.openjdk.org/jdk/commit/354c6605e32790ca421869636d8bf5456fc51717
Stats:     42 lines in 2 files changed: 16 ins; 11 del; 15 mod

8307185: pkcs11 native libraries make JNI calls into java code while holding GC lock

Reviewed-by: alanb, valeriep

-------------

PR: https://git.openjdk.org/jdk/pull/14931