RFR: Degenerated GC mode for Traversal

[Aleksey Shipilev]

On 03/27/2018 01:18 PM, Roman Kennke wrote:
>>  *) In service_concurrent_traversal_cycle, why do we need this check? I thought that coming out with
>> cancellation from final_traversal means we have the failure during degen, and it should upgrade to
>> Full GC right away?
>>
>>  275   if (heap->is_concurrent_traversal_in_progress()) {
>>  276     if (check_cancellation_or_degen(ShenandoahHeap::_degenerated_traversal)) return;
>>  277   }
> 
> We must not break out of the cycle between successful final-traversal
> and cleanup phases. This might happen because we're back into Java land
> by then, and allocations may fail before we managed to clean up. If this
> happens, we would get into degen with trashed regions. Do you have any
> suggestions how else to deal with that?

Ok, that sounds fair. But by this logic, we should not even have the entire block predicated by
is_concurrent_traversal_in_progress, right? Instead, we should go to cleanup unconditionally after
final-traversal. It would be the same thing we do for final-mark -> cleanup pair in normal cycle.

-Aleksey