RFR: 8332865: ubsan: os::attempt_reserve_memory_between reports overflow

[Thomas Stuefe]

On Tue, 4 Jun 2024 15:19:11 GMT, Matthias Baesken <mbaesken at openjdk.org> wrote:

> When running by ubsan-enabled binaries on Linux x86_64, os::attempt_reserve_memory_between reports overflows.
> 
> "runtime error: pointer index expression with base 0x000000001000 overflowed to 0xfffffffffffff000"
> 
> This coding triggers the ubsan issue
> 
> 
>   char* const hi_att = align_down(MIN2(max, absolute_max) - bytes, alignment_adjusted);
>   if (hi_att > max) {
>     return nullptr; // overflow
>   }
> 
> 
> However the function already contains overflow handling, so probably it is sufficient to add an attribute to the function os::attempt_reserve_memory_between to disable ubsan checks for this function.

Hmm, this is partly my fault. It's my code, and Matthias asked me before whether I could spot any error, which I could not, so I adviced him to mute ubsan.

I added the callstack Matthias sent me to the JBS issue. 

Looking closely, the error line number makes no sense. Maybe it was not in mainline, but an older JDK version?

This function is called in a gtest (attempt_reserve_memory_between_combos) that tries every possible combination (even very unlikely) of range boundaries,alignments,allocation size.  

ubsan says that the pointer that overflowed originally was 0x1000. The only address that can be 0x1000 around the failing line number is `min`. The lowest value for `max` is 0x2000 (`min` + smallest range size 0x1000).

---

@MBaesken : Can you please:

- doublecheck the ubsan error line?
- reproduce the problem, run the gtestlauncher manually and give us the output of`-Xlog:os+map=debug`? (You can just pass vm options to the gtestlauncher). That way, we know exactly which parameters the function was called with.

Also, on the box where we see the error, can you please add the print out for `sysctl vm.mmap_min_addr`?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/19543#issuecomment-2148947084