SoftReferences not cleared before OOME

[Devin Smith]

Hi,

This is in reference to an issue I posted on
https://github.com/adoptium/adoptium-support/issues/1096; it was suggested
I post to this list.

I'm able to reliably reproduce an issue whereby the JVM throws an
OutOfMemoryException where the majority of heap space is reclaimable soft
references. This can be verified with `-XX:+HeapDumpOnOutOfMemoryError` and
analyzing the resulting dump in VisualVM (or other heap dump tools). The
condition is triggered by also having concurrent computation inside JNI
critical regions (subject to multiple attempts wrt
`-XX:GCLockerRetryAllocationCount`).

```

[1.242s][warning][gc,alloc] main: Retried waiting for GCLocker too
often allocating 524290 words
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:71)
        at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:391)
        at io.deephaven.example.GCLockerTooOftenAllocating.main(GCLockerTooOftenAllocating.java:45)
```


This behavior seems to go against one of the core tenants of memory
management / SoftReference docs:

> All soft references to softly-reachable objects are guaranteed to have
been cleared before the virtual machine throws an OutOfMemoryError

I'm able to reproduce this across all LTS versions with a variety of GC
collectors; it appears that ZGC and Shenandoah do *not* exhibit this
behavior. Possibly b/c ZGC has already fixed this bug
https://bugs.openjdk.org/browse/JDK-8289838 (there are a number of OpenJDK
issues with similar titles).

https://github.com/devinrsmith/GCLockerTooOftenAllocating provides the
example code necessary to reproduce.

Thanks,
-Devin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/hotspot-gc-use/attachments/20240523/4bbe23cb/attachment.htm>