RFR: 8332865: ubsan: os::attempt_reserve_memory_between reports overflow

[Thomas Stuefe]

On Tue, 4 Jun 2024 15:19:11 GMT, Matthias Baesken <mbaesken at openjdk.org> wrote:

> When running by ubsan-enabled binaries on Linux x86_64, os::attempt_reserve_memory_between reports overflows.
> This happens in the  :tier1 tests ( gtest/LargePageGtests_use-large-pages.jtr )
> 
> 
> "runtime error: pointer index expression with base 0x000000001000 overflowed to 0xfffffffffffff000"
> 
> This coding triggers the ubsan issue
> 
> 
>   char* const hi_att = align_down(MIN2(max, absolute_max) - bytes, alignment_adjusted);
>   if (hi_att > max) {
>     return nullptr; // overflow
>   }
> 
> 
> However the function already contains overflow handling, so probably it is sufficient to add an attribute to the function os::attempt_reserve_memory_between to disable ubsan checks for this function.

Okay, I re-run the ubsan test manually on my local x64 machine. With logging, and after squashing about a zillion unrelated ubsan errors, I see:


[0,311s][debug][os,map] reserve_between (range [0x0000000000000000-0x0000000000001000), size 0x2000, alignment 0x1000, randomize: 1)
/shared/projects/openjdk/jdk-jdk/source/src/hotspot/share/runtime/os.cpp:1938:34: runtime error: pointer index expression with base 0x000000001000 overflowed to 0xfffffffffffff000
    #0 0x7f517dcda84e in os::attempt_reserve_memory_between(char*, char*, unsigned long, unsigned long, bool) /shared/projects/openjdk/jdk-jdk/source/src/hotspot/share/runtime/os.cpp:1938
    #1 0x7f517a83fbec in call_attempt_reserve_memory_between /shared/projects/openjdk/jdk-jdk/source/test/hotspot/gtest/runtime/test_os_reserve_between.cpp:69


So, the size of the mapping to be placed is larger than even the upper range boundary.

The fix is simple:


--- a/src/hotspot/share/runtime/os.cpp
+++ b/src/hotspot/share/runtime/os.cpp
@@ -1935,6 +1935,10 @@ char* os::attempt_reserve_memory_between(char* min, char* max, size_t bytes, siz
     return nullptr; // overflow
   }
 
+  char* const hi_end = MIN2(max, absolute_max);
+  if ((uintptr_t)hi_end < bytes) {
+    return nullptr; // no need to go on
+  }
   char* const hi_att = align_down(MIN2(max, absolute_max) - bytes, alignment_adjusted);
   if (hi_att > max) {
     return nullptr; // overflow

-------------

PR Comment: https://git.openjdk.org/jdk/pull/19543#issuecomment-2149805825