RFR: 8332865: ubsan: os::attempt_reserve_memory_between reports overflow

[Thomas Stuefe]

On Wed, 5 Jun 2024 08:00:04 GMT, Matthias Baesken <mbaesken at openjdk.org> wrote:

>> src/hotspot/share/runtime/os.cpp line 1894:
>> 
>>> 1892: #if defined(__clang__) || defined(__GNUC__)
>>> 1893: __attribute__((no_sanitize("undefined")))
>>> 1894: #endif
>> 
>> I don't think this attribute addition should be made. I agree with @stefank that we should be eliminating the
>> potential overflow, since the compiler (without ubsan) is within its rights to discard a pointer overflow check,
>> since pointer overflow is UB.
>
> From what I saw and got info from Thomas,  my impression was that the method contains already overflow handling.
> But if you think the current overflow handling is not 'good enough' , maybe it needs improvement.

Yeah, I am more confused now than I was initially :) If you provide us with the requested info, maybe we can shed light onto this one.

I know that I spend a lot of time with this code, making sure all corner cases are clean. Therefore I am curious what ubsan digged up.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/19543#discussion_r1627267370