Should OutOfMemoryError from NIO direct memory honor JVM flags for handling OOME?

[Alan Bateman]

On 26/09/2022 08:35, David Holmes wrote:
> :
>
> If all you really need is heap dumps then aren't there other ways to 
> get those via the management API (that you are already using)?
>
> I really do not want to turn these VM flags into an exported API for 
> use by JDK code if it can be avoided (they are crude and basic and 
> minimally tested). If that kind of functionality is needed then I'd 
> prefer to see a JDK-side API to provide it in a cleaner, more 
> supported way.

We added the built-in heap dumper and HeapDumpOnOutOfMemoryError in JDK 
6 to help with post mortem diagnosability of OOME. They have been 
supported and documented (e.g. troubleshooting guide) and there are 
several tools that support reading the .hprof format. It may be 
rudimentary but the option to enable it seems to be widely used.

Yes, you can use the management APIs and BufferPoolMXBean to monitor the 
use of direct memory and use the HotSpotDiagnosticMXBean "dumpHeap" 
operation to generate a heap dump at any time. You can use another 
HotSpotDiagnosticMXBean operation to enable the 
HeapDumpOnOutOfMemoryError at runtime but it's not going to trigger if 
the OOME is triggered trying to allocate direct memory.

I don't know if you have a JDK-specific API in mind but I think the 
discussion here is about having a way to trigger a heap dump (and maybe 
other OOME actions) when allocating direct memory fails.

-Alan