SoftReferences not cleared before OOME

[Devin Smith]

Hi,

This is a repost of
https://mail.openjdk.org/pipermail/hotspot-gc-use/2024-May/002938.html
; I'm elevating here as it seems like this list is more active.

I'm able to reliably reproduce an issue whereby the JVM throws an
OutOfMemoryException where the majority of heap space is reclaimable
soft references. This can be verified with
`-XX:+HeapDumpOnOutOfMemoryError` and analyzing the resulting dump in
VisualVM (or other heap dump tools). The condition is triggered by
also having concurrent computation inside JNI critical regions
(subject to multiple attempts wrt `-XX:GCLockerRetryAllocationCount`).

```
[1.242s][warning][gc,alloc] main: Retried waiting for GCLocker too
often allocating 524290 words
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:71)
        at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:391)
        at io.deephaven.example.GCLockerTooOftenAllocating.main(GCLockerTooOftenAllocating.java:45)
```

This behavior seems to go against one of the core tenants of memory
management / SoftReference docs:

> All soft references to softly-reachable objects are guaranteed to have been cleared before the virtual machine throws an OutOfMemoryError

I'm able to reproduce this across all LTS versions with a variety of
GC collectors; it appears that ZGC and Shenandoah do not exhibit this
behavior. Possibly b/c ZGC has already fixed this bug
https://bugs.openjdk.org/browse/JDK-8289838 (there are a number of
OpenJDK issues with similar titles).

https://github.com/devinrsmith/GCLockerTooOftenAllocating provides the
example code necessary to reproduce.

Is this an OpenJDK bug?

If so, would somebody with permissions elevate this to https://bugs.openjdk.org?

If it's not a bug, I'm hoping to gain some clarity on SoftReference /
OOME guarantees.

I'm happy to provide additional assistance if needed.

Thanks,
-Devin