RFR: JDK-8295889: NMT preinit code does not handle allocation errors [v2]

[Thomas Stuefe]

On Wed, 26 Oct 2022 05:45:33 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> The NMT preinit allocator (used for os::malloc and friends before the VM is initialized) does not handle malloc errors, nor does it handle overflows due to large sizes (it uses malloc headers too). Both cases need to be handled.
>> 
>> However, we can keep matters very simple. No need to propagate errors up to the caller; we can just fatal out on errors here since, in this phase, there is no alternative for failed allocations.
>
> Thomas Stuefe has updated the pull request incrementally with one additional commit since the last revision:
> 
>   assert overflows (debug+release)

> I'm more concerned that we don't know if this path may be influenced by external input. :(

The only randomness source I can think of is arguments. Argument handling is done before parsing and therefore before NMT initialization in preinit time. It influences the number of allocated blocks as well as their size. So if someone were to pass an argument close to size_max to the VM, it would run into overflow :-) but that is not possible, of course.

> These kind of "just in case" checks all add up.

True. But as this is security relevant, I'll leave it in. I rather shave off time somewhere else. E.g. in the real os::malloc, there are some optimizations we can do. E.g. getting rid of the atomic allocation counters in os.cpp since we do have NMT counting and malloc limits now. That is one thing I have on my list.

-------------

PR: https://git.openjdk.org/jdk/pull/10855