Should OutOfMemoryError from NIO direct memory honor JVM flags for handling OOME?

[Man Cao]

> If all you really need is heap dumps then aren't there other ways to get
> those via the management API (that you are already using)?

As Alan pointed out, for the direct memory OOME case, the timing of
triggering the heap dump is difficult without HeapDumpOnOutOfMemoryError.
If the direct memory API offers a callback that executes a user-provided
function on each allocation or before throwing OOME, that would be easier
for user code to trigger a heap dump before OOME.
However, supporting HeapDumpOnOutOfMemoryError and related flags seems much
easier than modifying the direct memory API.

> I really do not want to turn these VM flags into an exported API for use
> by JDK code if it can be avoided (they are crude and basic and minimally
> tested). If that kind of functionality is needed then I'd prefer to see
> a JDK-side API to provide it in a cleaner, more supported way.

Even if we provide a JDK-side API (like the callback idea mentioned above),
I suppose the API would not respect HeapDumpOnOutOfMemoryError and related
JVM flags?
As Alan (and [~stuefe] and [~xliu] from JDK-8257790) mentioned, those JVM
flags are well-known and users would expect them to work when they need to
debug these real resource-exhaustion OOMEs.

For the "crude and basic and minimally tested" argument, I don't think
it is a reason to not support them for these useful cases. These OOME flags
are extremely widely used in our production jobs (and possibly also for SAP
and Amazon). I'm confident the community will help maintain these flags and
make sure they function as expected.

-Man
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/nio-dev/attachments/20220926/0b485f15/attachment-0001.htm>