RFR: Degenerated GC mode for Traversal

[Aleksey Shipilev]

On 03/27/2018 01:46 PM, Roman Kennke wrote:
>>>> Ok, that sounds fair. But by this logic, we should not even have the entire block predicated by
>>>> is_concurrent_traversal_in_progress, right? Instead, we should go to cleanup unconditionally after
>>>> final-traversal. It would be the same thing we do for final-mark -> cleanup pair in normal cycle.
>>>
>>> If final-traversal *fails* (which happens before we trash regions) we
>>> actually want to dive into degen. Hence the check.
>>>
>>> I just had the idea that we might want to have a special degen-point for
>>> this situation: if we fail after successful traversal GC, but before
>>> cleanup, we might just want to finish cleanup under STW/degen, and then
>>> carry on. Then the check would look like:
>>>
>>> if (heap->is_concurrent_traversal_in_progress()) {
>>>   if
>>> (check_cancellation_or_degen(ShenandoahHeap::_degenerated_traversal))
>>> return;
>>> } else {
>>>   if
>>> (check_cancellation_or_degen(ShenandoahHeap::_degenerated_traversal_cleanup))
>>> return;
>>> }
>>>
>>> or such?
>>>
>>> WDYT?
>>>
>>
>> Like this:
>>
>> Diff:
>> http://cr.openjdk.java.net/~rkennke/traversal-degen/webrev.03.diff/
>> Full:
>> http://cr.openjdk.java.net/~rkennke/traversal-degen/webrev.03/
>>
> 
> OTOH, this is a matter of luck. If it runs OOM after the check, it will
> go to full-gc anyway?

Yes. So I am thinking that we can just slide to cleanup after final-traversal failure, and then the
allocation failure handling would restart degen-traversal itself from the known "clean" state? The
only difference against diving into STW straight away would be non-allocating mutators running while
alloc-failure is pending. Which seems to be no different from any other alloc-failure pending. No
need for separate degen point as well.

-Aleksey