RFR(S) 8056154: JVM crash with EXCEPTION_ACCESS_VIOLATION when there are many threads running

[Igor Veresov]

The main cause of this failure is that class redefinition operation may make all nmethods not entrant, including compiled MH intrinsics. After that ICs referencing those get relinked to c2i adapters that are going to call interpreter versions of MH intrinsics that, in turn, may all compiled methods, converting the same frame with i2c. And since c2i is not composable with i2c the problems ensue. 

The fix is to prevent compiled MH intrinsics from becoming not entrant. I also added a bunch of asserts that watch the invariant that a MH intrinsic should not exist without its compiled version (exception when running in pure interpreter) and compiledICs should not link with their c2i entries.

Webrev: http://cr.openjdk.java.net/~iveresov/8056154/webrev.00/

Thanks,
igor