RFR: 8332865: ubsan: os::attempt_reserve_memory_between reports overflow

[Stefan Karlsson]

On Tue, 4 Jun 2024 15:19:11 GMT, Matthias Baesken <mbaesken at openjdk.org> wrote:

> When running by ubsan-enabled binaries on Linux x86_64, os::attempt_reserve_memory_between reports overflows.
> 
> "runtime error: pointer index expression with base 0x000000001000 overflowed to 0xfffffffffffff000"
> 
> This coding triggers the ubsan issue
> 
> 
>   char* const hi_att = align_down(MIN2(max, absolute_max) - bytes, alignment_adjusted);
>   if (hi_att > max) {
>     return nullptr; // overflow
>   }
> 
> 
> However the function already contains overflow handling, so probably it is sufficient to add an attribute to the function os::attempt_reserve_memory_between to disable ubsan checks for this function.

Why is this better than fixing the overflow that causes the undefined behavior? IIUC, the current overflow checks is causing UB and that allows the compiler to do whatever, for example skip the return?

-------------

PR Review: https://git.openjdk.org/jdk/pull/19543#pullrequestreview-2097294034