RFR: 8335126: Shenandoah: Improve OOM handling

[Kelvin Nilsen]

On Thu, 27 Jun 2024 18:50:28 GMT, Y. Srinivas Ramakrishna <ysr at openjdk.org> wrote:

>> I think this is OK for now because we expect to integrate generational shenandoah soon, but am fine with defering it to GenShen integration in the future if other reviewers feel that would be better.
>> 
>> One question I had was about the policy for the treatment/reclamation of soft references, and whether that changes and/or escalates to "clear all" upon reaching a "less than good progress" state prior to an OOM (reflecting the best efforts promise prior to OOM).
>
> My soft ref clearing policy question above was cleared up in the code walkthrough today. We do clear all soft refs upon an allocation failure and therefore, _a fortiori_, before throwing an OOM.

In ShenanoahControlThread::run_service(), we set set_should_clear_all_soft_refs(true) if alloc_failure_pending.  This has the effect of expediting soft-ref processing even before we escalate to full gc.  ShenandoahFullGC::phase1_mark_heap() also sets soft_reference_policy(true), to "forcefully purge all soft references".

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/19912#discussion_r1657678985