[jdk16] RFR: 8259063: Possible deadlock with vtable/itable creation vs concurrent class unloading [v2]

[Nils Eliasson]

On Tue, 12 Jan 2021 15:28:19 GMT, Erik Österlund <eosterlund at openjdk.org> wrote:

>> There is an offender of consistent lock ordering: the creation of itable/vtable stubs. It happens when an inline cache is transitioning to megamorphic. Way earlier in the call stack, we grab the nmethod lock for IC patching, and then when a megamorphic transition happens *and* there is no stub created yet for the given vtable/itable index (rather uncommon), a new vtable/itable stub is created. This creation of a stub takes the CodeCache lock. We already deal with failure to create the stubs today, by making the inline cache "clean" instead, essentially deferring the transition to megamorphic to a subsequent call. We deal with that today to handle the code cache running out of memory, making it temporarily impossible to transition to megamorphic until memory has been freed up. My patch rides on that logic, so that we can try_lock() the CodeCache_lock instead. If we fail to take the lock, then it is arguably a bad time to do this megamorphic transition, regardless of deadlocks, as w
 e might have to wait quite a while. So if we fail to take the lock, I just return NULL saying we couldn't create a vtable/itable stub at this time, signalling we defer the transition to later, as we would also do if we couldn't create the stub due to memory exhaustion. This solves the deadlock situation.
>> 
>> Eric Caspole who reported the bug has tried this patch, and it solved the problem. I also tried it with the same test myself, with the same successful results. I also ran it through tier1-5 as it is touching inline cache code which can be a bit subtle.
>
> Erik Österlund has updated the pull request incrementally with one additional commit since the last revision:
> 
>   don't handle code cache exhaustion fully during inline cache patching

I like the solution - simple and without adding extra complexity or lock juggling.

It's a good just skipping the call to handle_full_code_cache. The VTableBlobs are not the thing usually filling up the code cache - and they can be placed in any codeheap. And even if we skip it - no harm done - there will be other allocations that will trigger the call.

-------------

Marked as reviewed by neliasso (Reviewer).

PR: https://git.openjdk.java.net/jdk16/pull/103