[9] RFR(S): 8152947: VM crash with assert(!removed || is_in_use()) failed: unused osr nmethod should be invalidated

[Tobias Hartmann]

Hi,

please review the following patch:

https://bugs.openjdk.java.net/browse/JDK-8152947
http://cr.openjdk.java.net/~thartmann/8152947/webrev.00/

JDK-8023191 added an assert to check if osr nmethods are invalidated (i.e. removed from the osr list). The assert is too strong because nmethod::invalidate_osr_method() may be executed by multiple threads that compete for making an osr nmethod not-entrant:

Thread 1: Invalidates the osr nmethod and blocks just before executing the assert in line 1397.
Thread 2: nmethod is already invalidated. Acquires patching lock and sets the nmethod to not-entrant.
Thread 1: Continues execution and hits the assert because the nmethod was on the list and is (now) not-entrant.

Thread 1 should ignore this and return. See also comment in nmethod::make_not_entrant_or_zombie():
  // another thread already performed this transition so nothing
  // to do, but return false to indicate this.

I changed the verification code to check for osr invalidation *after* the state of the nmethod was set.

Tested with failing test, JPRT and RBT (in progress).

Thanks,
Tobias