[jdk16] RFR: 8259063: Possible deadlock with vtable/itable creation vs concurrent class unloading

[Per Liden]

On Mon, 11 Jan 2021 16:11:34 GMT, Erik Österlund <eosterlund at openjdk.org> wrote:

> There is an offender of consistent lock ordering: the creation of itable/vtable stubs. It happens when an inline cache is transitioning to megamorphic. Way earlier in the call stack, we grab the nmethod lock for IC patching, and then when a megamorphic transition happens *and* there is no stub created yet for the given vtable/itable index (rather uncommon), a new vtable/itable stub is created. This creation of a stub takes the CodeCache lock. We already deal with failure to create the stubs today, by making the inline cache "clean" instead, essentially deferring the transition to megamorphic to a subsequent call. We deal with that today to handle the code cache running out of memory, making it temporarily impossible to transition to megamorphic until memory has been freed up. My patch rides on that logic, so that we can try_lock() the CodeCache_lock instead. If we fail to take the lock, then it is arguably a bad time to do this megamorphic transition, regardless of deadlocks, as we
  might have to wait quite a while. So if we fail to take the lock, I just return NULL saying we couldn't create a vtable/itable stub at this time, signalling we defer the transition to later, as we would also do if we couldn't create the stub due to memory exhaustion. This solves the deadlock situation.
> 
> Eric Caspole who reported the bug has tried this patch, and it solved the problem. I also tried it with the same test myself, with the same successful results. I also ran it through tier1-5 as it is touching inline cache code which can be a bit subtle.

Looks good to me!

-------------

Marked as reviewed by pliden (Reviewer).

PR: https://git.openjdk.java.net/jdk16/pull/103