RFR(S) 8056154: JVM crash with EXCEPTION_ACCESS_VIOLATION when there are many threads running

[Vladimir Kozlov]

Looks good.

compiledIC.cpp: compute_entry() should not be called when compilation is disabled (-Xint) UseCompiler=false. It should 
be called only from compiled code.
On other hand UseCompiler could be set to 'false' when codecache is full. So I am not sure that you should relax assert 
by checking !UseCompiler.
Or explain why you need it.

Otherwise it is good. Thank you for finding the cause!

Thanks,
Vladimir

On 9/4/14 9:24 PM, Igor Veresov wrote:
> The main cause of this failure is that class redefinition operation may make all nmethods not entrant, including compiled MH intrinsics. After that ICs referencing those get relinked to c2i adapters that are going to call interpreter versions of MH intrinsics that, in turn, may all compiled methods, converting the same frame with i2c. And since c2i is not composable with i2c the problems ensue.
>
> The fix is to prevent compiled MH intrinsics from becoming not entrant. I also added a bunch of asserts that watch the invariant that a MH intrinsic should not exist without its compiled version (exception when running in pure interpreter) and compiledICs should not link with their c2i entries.
>
> Webrev: http://cr.openjdk.java.net/~iveresov/8056154/webrev.00/
>
> Thanks,
> igor
>