RFR: 8339242: Fix overflow issues in AdlArena [v3]

[Kim Barrett]

On Tue, 3 Sep 2024 13:46:57 GMT, Casper Norrbin <duke at openjdk.org> wrote:

>> Hi everyone,
>> 
>> This PR addresses an issue in `adlArena` where some allocations lack checks for overflow. This could potentially result in successful allocations when called with unrealistic values.
>> 
>> The fix includes:
>> 
>> - Adding assertions to check for potential overflow.
>> - Reordering some operations to guard against overflow.
>
> Casper Norrbin has updated the pull request incrementally with one additional commit since the last revision:
> 
>   saturated pointer adds + size asserts

I've only commented on the adlc changes, but I think all of those comments also
apply to Arena.

src/hotspot/share/adlc/adlArena.cpp line 156:

> 154:   if (new_size <= old_size) {     // Shrink in-place
> 155:     if (c_old + old_size == _hwm) // Attempt to free the excess bytes
> 156:       _hwm = c_old + new_size;    // Adjust hwm

This `if` is missing braces around the consequent.  If fixing the whitespace, the missing braces should
also be added.

src/hotspot/share/adlc/adlArena.hpp line 107:

> 105:   // Fast allocate in the arena.  Common case is: pointer test + increment.
> 106:   void* Amalloc(size_t x) {
> 107:   assert(x <= (size_t)1 << 31, "unreasonable arena allocation size");

This seems mostly unrelated to the other overflow avoidance checking, and I think doesn't belong in
this PR.  For one thing, this might not be the right mechanism.  Perhaps an Arena should have a
configurable max allocation size?  And shouldn't the max allocation be applied to Arealloc too?

-------------

Changes requested by kbarrett (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/20774#pullrequestreview-2281964035
PR Review Comment: https://git.openjdk.org/jdk/pull/20774#discussion_r1744889478
PR Review Comment: https://git.openjdk.org/jdk/pull/20774#discussion_r1744886811