RFR: 8371048: ImageFileReader::open fails to check return value of osSupport::map_memory

[David Beaumont]

On Fri, 31 Oct 2025 14:00:38 GMT, Justin King <jcking at openjdk.org> wrote:

> Check whether `osSupport::map_memory` actually succeeded in all compliation modes, instead of crashing shortly after in non-debug builds. Ideally we should fall back to just reading the entire file into memory manually or use seek+read, but this is good enough for now to avoid crashing.

Returning false here isn't an immediate hard-fail (unlike the assertion), so there is now a situation where `mmap()` failing will allow the VM startup code to continue running for longer than before.

In particular, having this return false means that `ClassLoader::lookup_vm_options()` returns false, and skips the parsing of options from the jimage file.

I'm not 100% clear if it would later fail, or just attempt to run in "exploded" mode, possibly yielding an unexpected/undefined state.

There are two methods in `ClassPathImageEntry` related to this:

`ClassPathImageEntry::jimage_non_null()` which will assert that the image was opened.
`ClassPathImageEntry::jimage()` which will not.

Depending on who calls these, and in what order, the JVM startup might now reach code it wouldn't have before when/if `mmap()` failed.

However this is no different to other ways in which the jimage open code can return false (esp. just not having the file there) so I don't think this, slight, change in possible behaviour incurs any more risk of getting into an odd state than was already present.

-------------

PR Review: https://git.openjdk.org/jdk/pull/28087#pullrequestreview-3459404369