RFR: 8339242: Fix overflow issues in AdlArena [v2]

[Kim Barrett]

On Mon, 2 Sep 2024 09:36:53 GMT, Casper Norrbin <duke at openjdk.org> wrote:

>> Hi everyone,
>> 
>> This PR addresses an issue in `adlArena` where some allocations lack checks for overflow. This could potentially result in successful allocations when called with unrealistic values.
>> 
>> The fix includes:
>> 
>> - Adding assertions to check for potential overflow.
>> - Reordering some operations to guard against overflow.
>
> Casper Norrbin has updated the pull request incrementally with one additional commit since the last revision:
> 
>   arena realloc overflow check

Changes requested by kbarrett (Reviewer).

src/hotspot/share/adlc/adlArena.cpp line 154:

> 152:   if( (c_old+old_size == _hwm) &&            // Adjusting recent thing
> 153:       ((size_t)(_max-c_old) >= new_size) ) { // Still fits where it sits, safe from overflow
> 154: 

It appears that this change isn't worrying about bad `old_ptr` or `old_size`
arguments, which is fine.  But the code can be further improved by replacing
lines 144-157 with something like

// Reallocating the most recent allocation?
if ((c_old + old_size) == _hwm) {
  assert(_chunk->bottom() <= c_old, "invariant");
  // Reallocate in place if it fits.  This also handles shrinking.
  if (pointer_delta(_max, c_old) >= new_size) {
    _hwm = c_old + new_size;
    return c_old;
  }
}

Of course, in adlc you can't use HotSpot's pointer_delta utility, so there
you'll need to use something like what's in the PR for that calculation.

Any check for an "unreasonable" size should happen in Amalloc, not here.

-------------

PR Review: https://git.openjdk.org/jdk/pull/20774#pullrequestreview-2276975396
PR Review Comment: https://git.openjdk.org/jdk/pull/20774#discussion_r1741784871