RFR 8027434: "-XX:OnOutOfMemoryError" uses fork instead of vfork

[David Holmes]

Hi Florian,

On 27/09/2018 3:08 PM, Florian Weimer wrote:
> * David Holmes:
> 
>> I think you may be missing my point. We take a signal that will
>> terminate the VM, and from the signal handler context start the error
>> reporting and as part of that try to fork_and_exec the onError command
>> requested by the user. My recollection from reading all this stuff is
>> that vfork may not be safe to call from a signal context.
> 
> On GNU/Linux, fork is not safe, either:
> 
>    <https://access.redhat.com/articles/2921161>
> 
> This is particularly visible if you've got a SIGSEGV handler that is
> called from malloc after heap corruption.  Then fork will most likely
> hang due to a self-deadlock.
> 
> vfork might actually be safer in this context because it does not run
> any fork handlers, but only if you take the required measures to use
> it safely.  On the other hand, the rest of the process will keep
> running, so you don't get the snapshot functionality that comes from
> fork.
> 
> In my opinion (reflected in the cited article), crash handlers should
> live outside the process, at least on Linux.
> 
>> That's why I suggested we only switch this for the OnOutOfMemoryError
>> case as it's not (normally?) executed from a signal context.
> 
> If it's a native OnOutOfMemoryError (is there such a thing?), then
> both vfork and fork are likely to fail themselves (but fork is more
> likely to do so).

AFAICS this only for Java OOM:

./share/oops/objArrayKlass.cpp: 
report_java_out_of_memory("Requested array size exceeds VM limit");
./share/oops/instanceKlass.cpp:    report_java_out_of_memory("Requested 
array size exceeds VM limit");
./share/oops/typeArrayKlass.cpp: 
report_java_out_of_memory("Requested array size exceeds VM limit");
./share/oops/arrayKlass.cpp:    report_java_out_of_memory("Requested 
array size exceeds VM limit");
./share/gc/shared/memAllocator.cpp:    report_java_out_of_memory("Java 
heap space");
./share/gc/shared/memAllocator.cpp:    report_java_out_of_memory("GC 
overhead limit exceeded");

So even if fork is as unsafe as vfork I'm not motivated to make a change 
("if it ain't [visibly] broke ...") in the signal case.

Thanks,
David