RFR: 8335126: Shenandoah: Improve OOM handling

[William Kemper]

Mostly clean backport. This commit prevents tests that try to exhaust the heap from running continuous back-to-back full GCs (resulting in timeouts).

-------------

Commit messages:
 - Backport 3a87eb5c4606ce39970962895315567e8606eba7

Changes: https://git.openjdk.org/shenandoah-jdk21u/pull/69/files
  Webrev: https://webrevs.openjdk.org/?repo=shenandoah-jdk21u&pr=69&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8335126
  Stats: 33 lines in 3 files changed: 13 ins; 1 del; 19 mod
  Patch: https://git.openjdk.org/shenandoah-jdk21u/pull/69.diff
  Fetch: git fetch https://git.openjdk.org/shenandoah-jdk21u.git pull/69/head:pull/69

PR: https://git.openjdk.org/shenandoah-jdk21u/pull/69