RFR: Degenerated GC mode for Traversal

[Roman Kennke]

Am 27.03.2018 um 13:21 schrieb Aleksey Shipilev:
> On 03/27/2018 01:18 PM, Roman Kennke wrote:
>>>  *) In service_concurrent_traversal_cycle, why do we need this check? I thought that coming out with
>>> cancellation from final_traversal means we have the failure during degen, and it should upgrade to
>>> Full GC right away?
>>>
>>>  275   if (heap->is_concurrent_traversal_in_progress()) {
>>>  276     if (check_cancellation_or_degen(ShenandoahHeap::_degenerated_traversal)) return;
>>>  277   }
>>
>> We must not break out of the cycle between successful final-traversal
>> and cleanup phases. This might happen because we're back into Java land
>> by then, and allocations may fail before we managed to clean up. If this
>> happens, we would get into degen with trashed regions. Do you have any
>> suggestions how else to deal with that?
> 
> Ok, that sounds fair. But by this logic, we should not even have the entire block predicated by
> is_concurrent_traversal_in_progress, right? Instead, we should go to cleanup unconditionally after
> final-traversal. It would be the same thing we do for final-mark -> cleanup pair in normal cycle.

If final-traversal *fails* (which happens before we trash regions) we
actually want to dive into degen. Hence the check.

I just had the idea that we might want to have a special degen-point for
this situation: if we fail after successful traversal GC, but before
cleanup, we might just want to finish cleanup under STW/degen, and then
carry on. Then the check would look like:

if (heap->is_concurrent_traversal_in_progress()) {
  if
(check_cancellation_or_degen(ShenandoahHeap::_degenerated_traversal))
return;
} else {
  if
(check_cancellation_or_degen(ShenandoahHeap::_degenerated_traversal_cleanup))
return;
}

or such?

WDYT?

Roman