Trouble with LRB cancelling GC and safepoints

[Kemper, William]

During testing, we hit an assert caused when a mutator thread failed to evacuate an object and tried to cancel gc:

assert(thread->thread_state() == _thread_in_vm, "coming from wrong thread state");

We reckon this is because the JRT_LEAF macro used for the load reference barrier entry point (shenandoahRuntime.cpp) does not transition threads to the _thread_in_vm state. Threads which enter by means of JRT_LEAF should not take safepoints, hold locks, allocate on the heap, etc. (at least, according to the comments in interfaceSupport.inline.hpp). We tried to fix the issue here: https://github.com/openjdk/shenandoah/pull/117?- but this has just given us a new assert because JRT_LEAF sets up the NoSafepointVerifier and this is directly incompatible with ThreadBlockInVM:


  if (_no_safepoint_count > 0) {
    print_owned_locks();
    assert(false, "Possible safepoint reached by thread that does not allow it");
  }

Do we really need to take a safepoint in ShenandoahHeap::try_cancel_gc? If so, should we use JRT_ENTRY instead of JRT_LEAF? This would mean changing the JIT call sites to pass in the current thread to the mutator LRB.

Thanks,
William