RFR[13]: 8227260: Can't deal with SharedRuntime::handle_wrong_method triggering more than once for interpreter calls

[Erik Österlund]

Hi,

The i2c adapter sets a thread-local "callee_target" Method*, which is 
caught (and cleared) by SharedRuntime::handle_wrong_method if the i2c 
call is "bad" (e.g. not_entrant). This error handler forwards execution 
to the callee c2i entry. If the SharedRuntime::handle_wrong_method 
method is called again due to the i2c2i call being still bad, then we 
will crash the VM in the following guarantee in 
SharedRuntime::handle_wrong_method:

Method* callee = thread->callee_target();
guarantee(callee != NULL && callee->is_method(), "bad handshake");

Unfortunately, the c2i entry can indeed fail again if it, e.g., hits the 
new class initialization entry barrier.

I think a solution to this problem should stop making assumptions about 
how many things can go wrong when calling a method from the interpreter.

I caught this in ZGC where the timing window for hitting this issue 
seems to be wider due to concurrent code cache unloading. But it is 
equally problematic for all GCs.
With ZGC, I could catch this failing in SPECjbb2015 where a static 
method is called from JNI. I could reliably (25% chance) reproduce it, 
and with the patch it no longer reproduces after 25 runs.
I also tried hs-tier1-5, and it looked good.

Webrev:
http://cr.openjdk.java.net/~eosterlund/8227260/webrev.00/

Bug:
https://bugs.openjdk.java.net/browse/JDK-8227260

Thanks,
/Erik