RFR: 8332865: ubsan: os::attempt_reserve_memory_between reports overflow

[Matthias Baesken]

On Tue, 4 Jun 2024 15:19:11 GMT, Matthias Baesken <mbaesken at openjdk.org> wrote:

> When running by ubsan-enabled binaries on Linux x86_64, os::attempt_reserve_memory_between reports overflows.
> 
> "runtime error: pointer index expression with base 0x000000001000 overflowed to 0xfffffffffffff000"
> 
> This coding triggers the ubsan issue
> 
> 
>   char* const hi_att = align_down(MIN2(max, absolute_max) - bytes, alignment_adjusted);
>   if (hi_att > max) {
>     return nullptr; // overflow
>   }
> 
> 
> However the function already contains overflow handling, so probably it is sufficient to add an attribute to the function os::attempt_reserve_memory_between to disable ubsan checks for this function.

I  rebuilt  with current sources from this morning. Here is the stack from this

src/hotspot/share/runtime/os.cpp:1938:34: runtime error: pointer index expression with base 0x000000001000 overflowed to 0xfffffffffffff000
    #0 0x7fe1b98d8c56 in os::attempt_reserve_memory_between(char*, char*, unsigned long, unsigned long, bool) src/hotspot/share/runtime/os.cpp:1938
    #1 0x7fe1b6c52620 in call_attempt_reserve_memory_between test/hotspot/gtest/runtime/test_os_reserve_between.cpp:69
    #2 0x7fe1b6c58e15 in test_attempt_reserve_memory_between test/hotspot/gtest/runtime/test_os_reserve_between.cpp:108
    #3 0x7fe1b6c5a3c3 in os_attempt_reserve_memory_between_combos_vm_Test::TestBody() test/hotspot/gtest/runtime/test_os_reserve_between.cpp:291
    #4 0x7fe1ba961603 in testing::Test::Run() /sapmnt/sapjvm_work/openjdk/tools/gtest/googletest-1.14.0/googletest/src/gtest.cc:2687
    #5 0x7fe1ba961603 in testing::Test::Run() /sapmnt/sapjvm_work/openjdk/tools/gtest/googletest-1.14.0/googletest/src/gtest.cc:2677
    #6 0x7fe1ba961d0d in testing::TestInfo::Run() /sapmnt/sapjvm_work/openjdk/tools/gtest/googletest-1.14.0/googletest/src/gtest.cc:2836
    #7 0x7fe1ba992618 in testing::TestSuite::Run() /sapmnt/sapjvm_work/openjdk/tools/gtest/googletest-1.14.0/googletest/src/gtest.cc:3015
    #8 0x7fe1ba992618 in testing::TestSuite::Run() /sapmnt/sapjvm_work/openjdk/tools/gtest/googletest-1.14.0/googletest/src/gtest.cc:2968
    #9 0x7fe1ba9935e5 in testing::internal::UnitTestImpl::RunAllTests() /sapmnt/sapjvm_work/openjdk/tools/gtest/googletest-1.14.0/googletest/src/gtest.cc:5920
    #10 0x7fe1ba95988f in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /sapmnt/sapjvm_work/openjdk/tools/gtest/googletest-1.14.0/googletest/src/gtest.cc:2670
    #11 0x7fe1ba95988f in testing::UnitTest::Run() /sapmnt/sapjvm_work/openjdk/tools/gtest/googletest-1.14.0/googletest/src/gtest.cc:5484
    #12 0x7fe1b65e4d27 in RUN_ALL_TESTS() /sapmnt/sapjvm_work/openjdk/tools/gtest/googletest-1.14.0/googletest/include/gtest/gtest.h:2317
    #13 0x7fe1b65e4d27 in runUnitTestsInner test/hotspot/gtest/gtestMain.cpp:290
    #14 0x56268b2d17d8 in main test/hotspot/gtest/gtestLauncher.cpp:40
    #15 0x7fe1b064624c in __libc_start_main (/lib64/libc.so.6+0x3524c) (BuildId: f732026552f6adff988b338e92d466bc81a01c37)

-------------

PR Comment: https://git.openjdk.org/jdk/pull/19543#issuecomment-2149655999