RFR: 8268163: Change the order of fallback full GCs in G1

[Thomas Schatzl]

On Thu, 3 Jun 2021 18:25:00 GMT, Stefan Johansson <sjohanss at openjdk.org> wrote:

> Please review this change to make the order of G1 Full GCs a bit more straight forward.
> 
> **Summary**
> In [JDK-8233822](https://bugs.openjdk.java.net/browse/JDK-8233822) the way Full GCs were scheduled was changed a bit to support a use-case introduced by [JDK-8202286](https://bugs.openjdk.java.net/browse/JDK-8202286) (which allowed the heap to have the old generation on an alternative memory device). This feature has been removed ([JDK-8256181](https://bugs.openjdk.java.net/browse/JDK-8256181)), but the Full GC scheduling has not been reverted. This can lead to situations where we do three Full GCs in a row. Doing more than two seems a bit over the top, so this change more or less reverts back to the old behavior 
> * Young collection requesting to allocate when done will cause at most two Full GCs, the first will not clear any soft references and allow dead wood to be left. The second one, if still not able to satisfy the allocation will clear soft references and compact everything not allowing any dead wood.
> * For concurrent start collections and young collections not requesting any allocation, one Full GC will be scheduled if no regions were freed up by the initial collection.
> 
> A change compared to current behavior is that a concurrent collection started because of the metadata threshold not being met, will no longer be upgraded to a Full GC. This is still equal to how things were handled prior to [JDK-8233822](https://bugs.openjdk.java.net/browse/JDK-8233822).
> 
> **Testing**
> Tier 1-3, plus manual testing looking at output in near OOM situations.

Marked as reviewed by tschatzl (Reviewer).

-------------

PR: https://git.openjdk.java.net/jdk/pull/4342