RFR: 8316653: Large NMethodSizeLimit triggers assert during C1 code buffer allocation [v2]

[Roberto Castañeda Lozano]

On Wed, 15 Nov 2023 09:19:46 GMT, Daniel Lundén <duke at openjdk.org> wrote:

>> This changeset fixes an overflow issue for the develop flag `-XX:NMethodSizeLimit`.
>> 
>> In summary, `java -XX:NMethodSizeLimit=2147483647` triggers an assert in `CodeCache::allocate`:
>> 
>> #  assert(size > 0) failed: Code cache allocation request must be > 0 but is -1932735136
>> 
>> Why?
>> 1. The function `code_buffer_size` computes `NMethodSizeLimit + NMethodSizeLimit/10` as a signed integer (may overflow).
>> 2. The `BufferBlob::create` function takes `code_buffer_size()` as argument and uses it (among other things, such as `sizeof(BufferBlob)`) to compute an unsigned size.
>> 3. `BufferBlob::operator new` treats the unsigned size as signed (may overflow).
>> 
>> Changes:
>> - Change all data types in the above chain to `uint` and leave the flag range for `NMethodSizeLimit` untouched (`range(0, max_jint)`). This ensures no overflow.
>> - Add a new regression test.
>> 
>> ### Testing
>> Platforms: windows-x64, linux-x64, linux-aarch64, macosx-x64, macosx-aarch64.
>> - `tier1`
>> - HotSpot parts of `tier2` and `tier3`
>
> Daniel Lundén has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Update src/hotspot/share/code/codeCache.cpp
>   
>   Co-authored-by: Roberto Castañeda Lozano <robcasloz at users.noreply.github.com>

Marked as reviewed by rcastanedalo (Reviewer).

-------------

PR Review: https://git.openjdk.org/jdk/pull/16656#pullrequestreview-1731624438