RFR 8027434: "-XX:OnOutOfMemoryError" uses fork instead of vfork

[Florian Weimer]

* Thomas Stüfe:

> Incidentally, we have a kind-of-self-healing mechanism in error
> reporting, where error reporting is done in steps and if one steps
> takes too long it gets interrupted via signal and we continue with the
> next step. This mechanism still does not cover onError, but it could
> be made to do so. Wonder whether that would be useful.

That sounds like too much work for a vfork-based approach being
reliable.

You could call the clone system call (not the glibc wrapper) directly
and get a classic fork, but without fork handlers.  But the new
process will be in an iffy state as a result.  Recent glibc versions
have eliminated the PID cache, so at least that source of
clone-specific bugs is gone, but there are other things that will not
work.  Most async-signal-safe functions should work.

>> vfork might actually be safer in this context because it does not run
>> any fork handlers, but only if you take the required measures to use
>> it safely.  On the other hand, the rest of the process will keep
>> running, so you don't get the snapshot functionality that comes from
>> fork.
>>
>> In my opinion (reflected in the cited article), crash handlers should
>> live outside the process, at least on Linux.
>
> I do not think anyone uses this mechanism to get cores, or? The VM
> will print an hs-err file, then abort with core. Usually that core is
> still "close enough" to the crash cause to be useful - especially
> since the error handler never unwinds the stack.

The idea is to have a plugin in the crash handler mechanism that
extracts that data from the frozen process image, before the core file
is written to disk (which may never happen).  ABRT has plugins, but I
don't know if this part is pluggable.

Writing coredumps to disk is usually not an option, I agree.  Perhaps
coredumps could work in more cases if the Java heap was marked
MADV_DONTDUMP as an option.  Then it would only contain native memory
and of a more digestable size.  But that's getting off-topic.