RFR: Make object pinning safepoint aware

[Roman Kennke]

Am 15.10.2017 um 15:45 schrieb Zhengyu Gu:
>>>
>>>
>> Ok, this is strange indeed. The write barrier should ensure we get no 
>> cset objects in pinning. The only logical explanation is that evac 
>> failed because of OOM (you should see than in hs_err: it will tell 
>> you if we are in cancelled state). However, we must still pin the 
>> object/region. I guess we must ensure we wait in 
>> oom_during_evacuation() until all GC threads settled (we probably do 
>> the right thing already), and then carry on pinning the cset object 
>> (i.e. we need an exception in the checking code for when evac has 
>> been cancelled). Then we'd slide into full-gc where we would handle 
>> pinning correctly (unless the region isn't pinned anymore when we get 
>> there).
>
> Here this the complete hs_err: 
> http://cr.openjdk.java.net/~zgu/shenandoah/obj_pin_safepoint/hs_err_pid17356.log

And, as expected, it's a cancelled GC:

Heap:
Shenandoah Heap
  131072K total, 131072K committed, 91450K used
  16 x 8192K regions
Status: idle cancelled

Which means that we need to make sure that the failure path from the JNI 
critical blah function up to the full-gc must work, and that the pinned 
array never moves during that process. With a little luck it just means 
to add a || heap->is_cancelled_gc() or so to the checking code.

Roman