RFR: 8332865: ubsan: os::attempt_reserve_memory_between reports overflow

[Matthias Baesken]

On Wed, 5 Jun 2024 04:35:26 GMT, Kim Barrett <kbarrett at openjdk.org> wrote:

>> When running by ubsan-enabled binaries on Linux x86_64, os::attempt_reserve_memory_between reports overflows.
>> 
>> "runtime error: pointer index expression with base 0x000000001000 overflowed to 0xfffffffffffff000"
>> 
>> This coding triggers the ubsan issue
>> 
>> 
>>   char* const hi_att = align_down(MIN2(max, absolute_max) - bytes, alignment_adjusted);
>>   if (hi_att > max) {
>>     return nullptr; // overflow
>>   }
>> 
>> 
>> However the function already contains overflow handling, so probably it is sufficient to add an attribute to the function os::attempt_reserve_memory_between to disable ubsan checks for this function.
>
> src/hotspot/share/runtime/os.cpp line 1894:
> 
>> 1892: #if defined(__clang__) || defined(__GNUC__)
>> 1893: __attribute__((no_sanitize("undefined")))
>> 1894: #endif
> 
> I don't think this attribute addition should be made. I agree with @stefank that we should be eliminating the
> potential overflow, since the compiler (without ubsan) is within its rights to discard a pointer overflow check,
> since pointer overflow is UB.

>From what I saw and got info from Thomas,  my impression was that the method contains already overflow handling.
But if you think the current overflow handling is not 'good enough' , maybe it needs improvement.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/19543#discussion_r1627200740