RFR: 8351334: [ubsan] memoryReserver.cpp:552:60: runtime error: applying non-zero offset 1073741824 to null pointer

[Axel Boldt-Christmas]

On Thu, 28 Aug 2025 08:49:06 GMT, Axel Boldt-Christmas <aboldtch at openjdk.org> wrote:

>I also wonder how we got here with HeapBaseMinAddress == 0 as we set the value back to the default if it is lower when using compressed oops (in Arguments::set_heap_size()). And I think the default is 2G on all platforms.

I realise now that this is based on if we have set the max heap explicitly. 

So we can end up with 0 as the lower bound in `try_reserve_range` which will cause us to request "any address", which seems like another bug. (Making sure `aligned_heap_base_min_address != 0` fixes this here, but `try_reserve_range` should probably guard agains having an `attach_point` which is 0 or a `lowest_start` which is 0.)

While experimenting with the flags I notice we do not protect against overflow in the reservation code so setting a high `HeapBaseMinAddress` will crash. And it is not captured by `TestOptionsWithRanges` because it cannot understand our constraint function and will only try SIZE_MAX, but we only allow SIZE_MAX aligned down to our alignment. Regardless seems like this code is crawling with bugs.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/26955#issuecomment-3232761537