RFR: 8346866: [ASAN] memoryReserver.cpp reported applying non-zero offset to non-null pointer produced null pointer [v8]

[SendaoYan]

On Tue, 7 Jan 2025 15:02:07 GMT, Axel Boldt-Christmas <aboldtch at openjdk.org> wrote:

>> SendaoYan has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   convert attach_point to size_t ang then convert to char* again, to avoid ubsan report "produced null pointer" fail.
>
> I wonder if we could rewrite this where our attempts are more explicit. That is we do exactly the `num_attempts_to_try` and calculate the stepsize in such a way that we do not have problems with calculation addresses which are outside of the range.
> 
> Something along these lines. First I found it weird that the current implementation calculates the step size with the `+1` included when the range is inclusive. I instead divide it into the number of intervals and align it down so that the attatchment point calculation does not go outside the range `[lowest_start, highest_start]`. 
> ```C++
> ReservedSpace HeapReserver::Instance::try_reserve_range(char *highest_start,
>                                                         char *lowest_start,
>                                                         size_t attach_point_alignment,
>                                                         char *aligned_heap_base_min_address,
>                                                         char *upper_bound,
>                                                         size_t size,
>                                                         size_t alignment,
>                                                         size_t page_size) {
>   assert(is_aligned(highest_start, attach_point_alignment), "precondition");
>   assert(is_aligned(lowest_start, attach_point_alignment), "precondition");
> 
>   const size_t attach_range = highest_start - lowest_start;
>   const uint64_t num_attempts_possible = (attach_range / attach_point_alignment) + 1;
>   const uint64_t num_attempts_to_try   = MIN2((uint64_t)HeapSearchSteps, num_attempts_possible);
>   const uint64_t num_intervals = num_attempts_to_try - 1;
>   const size_t stepsize = num_intervals == 0 ? 0 : align_down(attach_range / num_intervals, attach_point_alignment);
> 
>   for (uint64_t i = 0; i < num_attempts_to_try; ++i) {
>     char* const attach_point = highest_start - stepsize * i;
>     ReservedSpace reserved = try_reserve_memory(size, alignment, page_size, attach_point);
> 
>     if (reserved.is_reserved()) {
>       if (reserved.base() >= aligned_heap_base_min_address &&
>           size <= (uintptr_t)(upper_bound - reserved.base())) {
>         // Got a successful reservation.
>         return reserved;
>       }
> 
>       release(reserved);
>     }
>   }
> 
>   // Failed
>   return {};
> }

> I like @xmas92 approach as it gets rid of the weird cast, type mixups, and (hopefully) solves the UB.

Thanks @xmas92 @stefank. The rewrite approach do solves the UB. Code has been updated. I will redo the additional test.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/22897#issuecomment-2575597884