RFR: 8364490: Fatal error on large SpecTrapLimitExtraEntries value

[Christian Hagedorn]

On Fri, 21 Nov 2025 12:39:01 GMT, Anton Seoane Ampudia <aseoane at openjdk.org> wrote:

> This PR addresses VM crashes on very large values for `SpecTrapLimitExtraEntries`.
> 
> The experimental `SpecTrapLimitExtraEntries` allows for a user-specified number of extra method data trap entries for speculation. Currently, this number is implemented with an `int`, which means that users can specify very large values that will translate into huge `MethodData` objects that cannot be allocated in Metaspace.
> 
> An `int` range of values should not be allowed, as negative `SpecTrapLimitExtraEntries` do not make any sense, and very high values (such as the ones that cause this crash) are equally nonsensical. This changeset adds a range to the flag values to address these issues.
> 
> `SpecTrapLimitExtraEntries` is `MAX`ed with HotSpot's computed heuristic, which means that in any case it can only serve as a buffer above the heuristic. Based on benchmarks where I logged heuristic-derived values for extra `DataLayout` cells, even a value of 50 for `SpecTrapLimitExtraEntries` is more than sufficient. To provide some headroom and keep things simple, I have set the upper limit to 100.
> 
> **Testing:** passes tiers 1-2

That looks reasonable to me. Can you add a sanity hello world test where we run with `SpecTrapLimitExtraEntries=0` and `SpecTrapLimitExtraEntries=100`? We do not seem to have any tests with that flag apart from one AOT test (`AOTProfileFlags.java`).

-------------

PR Review: https://git.openjdk.org/jdk/pull/28451#pullrequestreview-3504922243