RFR: 8332865: ubsan: os::attempt_reserve_memory_between reports overflow [v2]

[Thomas Stuefe]

On Wed, 5 Jun 2024 14:32:11 GMT, Matthias Baesken <mbaesken at openjdk.org> wrote:

>> When running by ubsan-enabled binaries on Linux x86_64, os::attempt_reserve_memory_between reports overflows.
>> This happens in the  :tier1 tests ( gtest/LargePageGtests_use-large-pages.jtr )
>> 
>> 
>> "runtime error: pointer index expression with base 0x000000001000 overflowed to 0xfffffffffffff000"
>> 
>> This coding triggers the ubsan issue
>> 
>> 
>>   char* const hi_att = align_down(MIN2(max, absolute_max) - bytes, alignment_adjusted);
>>   if (hi_att > max) {
>>     return nullptr; // overflow
>>   }
>> 
>> 
>> However the function already contains overflow handling, so probably it is sufficient to add an attribute to the function os::attempt_reserve_memory_between to disable ubsan checks for this function.
>
> Matthias Baesken has updated the pull request incrementally with two additional commits since the last revision:
> 
>  - reuse hi_end
>  - add check suggested by Thomas Stuefe

Looks good. If checks are green and ubsan is happy, good to go.

-------------

Marked as reviewed by stuefe (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/19543#pullrequestreview-2099435957