RFR: 8273608: Deadlock when jcmd of OnError attaches to itself

[Thomas Stuefe]

On Wed, 22 Sep 2021 07:38:22 GMT, Xin Liu <xliu at openjdk.org> wrote:

> > Can we limit this to the jcmd-attaches-to-me scenario? In general, the less we modify the VM state before core'ing the better. This distorts the picture and may confuse analysts of the hs-err file/core. I think we should do this only if necessary.
> > Potentially, I would even limit it to OOM situations since for other types of errors (eg crashes) I do not see the point of attaching with jcmd. To prevent deadlock in those cases, one may just avoid calling jcmd altogether.
> 
> The only reason I try this because I would like to get heap dump when `-XX:AbortVMOnException=java.lang.OutOfMemoryError ` does trigger a fatal.
> 
> Indeed, I know we can get a core file and extract java heap from it. Some counter-arguments are: 1) core dump is subject to kernel and ulimit constraints. 2) filesize is too big 3) not secure. I come up an idea to use OnError=jcmd %p GC.heap_dump to simulate `HeapDumpOnOutOfMemoryError`.
> 
> if neither of you guys thinks it's a good idea, I can drop it. As you said, it will distort VMThread for sure.

I am not against your fix. Makes sense and offers some merits compared to the usual core file analysis.

All I am saying is that I would limit it to the case of OnError="jcmd <my pid>". And maybe just to OOMs.

E.g. If someone does an OnError with a different tool or script, maybe something harmless like OnError="cp core /my-core-dir", I would rather we don't switch the thread to native.

Subject to discussion, oc. Maybe we could limit it to OOMs but leave it for all values of OnError.

(Its all a best-effort anyway. E.g. when OOMing due to thread creation error, the fork needed for OnError won't work either)

-------------

PR: https://git.openjdk.java.net/jdk/pull/5590