RFR: 8332865: ubsan: os::attempt_reserve_memory_between reports overflow [v2]

[Matthias Baesken]

> When running by ubsan-enabled binaries on Linux x86_64, os::attempt_reserve_memory_between reports overflows.
> This happens in the  :tier1 tests ( gtest/LargePageGtests_use-large-pages.jtr )
> 
> 
> "runtime error: pointer index expression with base 0x000000001000 overflowed to 0xfffffffffffff000"
> 
> This coding triggers the ubsan issue
> 
> 
>   char* const hi_att = align_down(MIN2(max, absolute_max) - bytes, alignment_adjusted);
>   if (hi_att > max) {
>     return nullptr; // overflow
>   }
> 
> 
> However the function already contains overflow handling, so probably it is sufficient to add an attribute to the function os::attempt_reserve_memory_between to disable ubsan checks for this function.

Matthias Baesken has updated the pull request incrementally with two additional commits since the last revision:

 - reuse hi_end
 - add check suggested by Thomas Stuefe

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/19543/files
  - new: https://git.openjdk.org/jdk/pull/19543/files/983bda19..71c91621

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=19543&range=01
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=19543&range=00-01

  Stats: 8 lines in 1 file changed: 4 ins; 3 del; 1 mod
  Patch: https://git.openjdk.org/jdk/pull/19543.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/19543/head:pull/19543

PR: https://git.openjdk.org/jdk/pull/19543