RFR: 8366925: Improper std::nothrow new expression in NativeHeapTrimmerThread ctor

[Kim Barrett]

On Mon, 15 Sep 2025 01:01:17 GMT, Guanqiang Han <ghan at openjdk.org> wrote:

> Please review this Patch.
> 
> **Description:**
> Add null check for _lock in NativeHeapTrimmerThread constructor. If allocation of the lock fails (_lock == nullptr), it indicates an out-of-memory condition. Since even this small allocation failed, it's unlikely that subsequent operations would succeed, so we terminate the JVM immediately.

Changes requested by kbarrett (Reviewer).

src/hotspot/share/runtime/trimNativeHeap.cpp line 174:

> 172:         vm_exit_out_of_memory(0, OOM_MALLOC_ERROR,
> 173:                               "Failed to allocate NativeHeapTrimmer_lock");
> 174:     }

There are a lot of places where, when creating a new thread in the VM, we try
to provide a soft landing when the creation fails (due to OOM or otherwise).
That tends to be more of a thing for asynchronously created VM threads, and
perhaps less so for those created at startup. I'm guessing this is one of the
latter. So I'm guessing the rationale for using the nothrow allocation of the
mutex was a botched attempt at providing such a soft landing. That code was in
the initial commit of the change that added this code, and there was no
discussion of it in the PR. On the other hand, the change author is available
to be asked: @tstuefe ?

It's also relevant that if the `os::create_thread` fails then we seem to just
quietly drop the whole tracking and trimming thing. To be consistent with that
one might keep the existing nothrow lock allocation and just change that
condition for starting the thread to first check for non-null `_lock`.

Basically, I think it should either try harder to not abort the application,
or should just do the simple thing of not using nothrow allocation for the
mutex. Adding code for a "better" error message for the abort doesn't seem
worthwhile to me.

-------------

PR Review: https://git.openjdk.org/jdk/pull/27275#pullrequestreview-3226040167
PR Review Comment: https://git.openjdk.org/jdk/pull/27275#discussion_r2349945663