RFR: 8318586: Explicitly handle upcall stub allocation failure

[Aleksey Shipilev]

On Mon, 23 Oct 2023 13:58:27 GMT, Jorn Vernee <jvernee at openjdk.org> wrote:

> Explicitly handle UpcallStub allocation failures by terminating. We currently might try to use the returned `nullptr` which would fail sooner or later. This patch just makes the termination explicit.

I find it pretty weird to terminate the VM if we cannot allocate upcall stub. Does this mean the user code could actually terminate the VM on this `fatal`? Unit test suggests so.

Can the VM code actually handle things without upcall stub present, if e.g. memory is exhausted?

src/hotspot/share/code/codeBlob.cpp line 761:

> 759:     MutexLocker mu(CodeCache_lock, Mutex::_no_safepoint_check_flag);
> 760:     blob = new (size) UpcallStub(name, cb, size, receiver, frame_data_offset);
> 761:     if (blob == nullptr) {

I think it would be safer to call into `fatal` without having `CodeCache_lock` held. Move it out of `MutexLocker` scope?

-------------

PR Review: https://git.openjdk.org/jdk/pull/16311#pullrequestreview-1696785943
PR Review Comment: https://git.openjdk.org/jdk/pull/16311#discussion_r1371421151