[jdk16] RFR: 8259063: Possible deadlock with vtable/itable creation vs concurrent class unloading [v2]

[Erik Österlund]

> There is an offender of consistent lock ordering: the creation of itable/vtable stubs. It happens when an inline cache is transitioning to megamorphic. Way earlier in the call stack, we grab the nmethod lock for IC patching, and then when a megamorphic transition happens *and* there is no stub created yet for the given vtable/itable index (rather uncommon), a new vtable/itable stub is created. This creation of a stub takes the CodeCache lock. We already deal with failure to create the stubs today, by making the inline cache "clean" instead, essentially deferring the transition to megamorphic to a subsequent call. We deal with that today to handle the code cache running out of memory, making it temporarily impossible to transition to megamorphic until memory has been freed up. My patch rides on that logic, so that we can try_lock() the CodeCache_lock instead. If we fail to take the lock, then it is arguably a bad time to do this megamorphic transition, regardless of deadlocks, as we
  might have to wait quite a while. So if we fail to take the lock, I just return NULL saying we couldn't create a vtable/itable stub at this time, signalling we defer the transition to later, as we would also do if we couldn't create the stub due to memory exhaustion. This solves the deadlock situation.
> 
> Eric Caspole who reported the bug has tried this patch, and it solved the problem. I also tried it with the same test myself, with the same successful results. I also ran it through tier1-5 as it is touching inline cache code which can be a bit subtle.

Erik Österlund has updated the pull request incrementally with one additional commit since the last revision:

  don't handle code cache exhaustion fully during inline cache patching

-------------

Changes:
  - all: https://git.openjdk.java.net/jdk16/pull/103/files
  - new: https://git.openjdk.java.net/jdk16/pull/103/files/71c09794..bdf75c79

Webrevs:
 - full: https://webrevs.openjdk.java.net/?repo=jdk16&pr=103&range=01
 - incr: https://webrevs.openjdk.java.net/?repo=jdk16&pr=103&range=00-01

  Stats: 19 lines in 4 files changed: 14 ins; 0 del; 5 mod
  Patch: https://git.openjdk.java.net/jdk16/pull/103.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk16 pull/103/head:pull/103

PR: https://git.openjdk.java.net/jdk16/pull/103