RFR: 8349554: [UBSAN] os::attempt_reserve_memory_between reported applying non-zero offset to non-null pointer produced null pointer

[SendaoYan]

Hi all,

Function 'os::attempt_reserve_memory_between(char*, char*, size_t, size_t, bool)' 'src/hotspot/share/runtime/os.cpp' reported "runtime error: applying non-zero offset to non-null pointer 0x000000001000 produced null pointer" by address sanitizer. Gtest in function 'os_attempt_reserve_memory_between_combos_vm_Test::TestBody'  at file test/hotspot/gtest/runtime/test_os_reserve_between.cpp call 'os::attempt_reserve_memory_between (min=0x0, max=0x1000, bytes=4096, alignment=4096, randomize=true)' trigger this failure. Before this PR, the pointer var `hi_end` get value from `max` 0x1000, and then apply offset `bytes`, and `max` equals `bytes`, thus address sanitizer report this failure.

This PR change the type of var `hi_end` from `char*` to `size_t` will eliminate the undefined behaviour, and do not change the original logic. Risk is low.

Additional testing:

- [ ] jtreg tests(which include tier1/2/3 etc.) on linux-x64
- [ ] jtreg tests(which include tier1/2/3 etc.) on linux-aarch64

-------------

Commit messages:
 - 8349554: [UBSAN] os::attempt_reserve_memory_between reported applying non-zero offset to non-null pointer produced null pointer

Changes: https://git.openjdk.org/jdk/pull/23508/files
  Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=23508&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8349554
  Stats: 3 lines in 1 file changed: 0 ins; 0 del; 3 mod
  Patch: https://git.openjdk.org/jdk/pull/23508.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/23508/head:pull/23508

PR: https://git.openjdk.org/jdk/pull/23508