Integrated: 8316653: Large NMethodSizeLimit triggers assert during C1 code buffer allocation

[Daniel Lundén]

On Tue, 14 Nov 2023 14:16:55 GMT, Daniel Lundén <duke at openjdk.org> wrote:

> This changeset fixes an overflow issue for the develop flag `-XX:NMethodSizeLimit`.
> 
> In summary, `java -XX:NMethodSizeLimit=2147483647` triggers an assert in `CodeCache::allocate`:
> 
> #  assert(size > 0) failed: Code cache allocation request must be > 0 but is -1932735136
> 
> Why?
> 1. The function `code_buffer_size` computes `NMethodSizeLimit + NMethodSizeLimit/10` as a signed integer (may overflow).
> 2. The `BufferBlob::create` function takes `code_buffer_size()` as argument and uses it (among other things, such as `sizeof(BufferBlob)`) to compute an unsigned size.
> 3. `BufferBlob::operator new` treats the unsigned size as signed (may overflow).
> 
> Changes:
> - Change all data types in the above chain to `uint` and leave the flag range for `NMethodSizeLimit` untouched (`range(0, max_jint)`). This ensures no overflow.
> - Add a new regression test.
> 
> ### Testing
> Platforms: windows-x64, linux-x64, linux-aarch64, macosx-x64, macosx-aarch64.
> - `tier1`
> - HotSpot parts of `tier2` and `tier3`

This pull request has now been integrated.

Changeset: c36ec2ca
Author:    Daniel Lundén <daniel.lunden at gmail.com>
Committer: Roberto Castañeda Lozano <rcastanedalo at openjdk.org>
URL:       https://git.openjdk.org/jdk/commit/c36ec2ca70248c2e4676fd725fbb132c3b929908
Stats:     28 lines in 8 files changed: 14 ins; 0 del; 14 mod

8316653: Large NMethodSizeLimit triggers assert during C1 code buffer allocation

Reviewed-by: kvn, rcastanedalo, thartmann

-------------

PR: https://git.openjdk.org/jdk/pull/16656