RFR: JDK-8318016: Per-compilation memory ceiling [v2]

[Roland Westrelin]

On Wed, 25 Oct 2023 19:28:59 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> This RFE introduces a new compile command to limit memory usage per compilation. It complements and is based upon the JIT memory statistic introduced with https://bugs.openjdk.org/browse/JDK-8317683.
>> 
>> This new memory limit is applied during compilation for either C1 or C2. It has one of two modes:
>> - *crash* mode (mainly for compiler devs), which aborts with hs-err file when the ceiling is reached, right in the compiler thread at the allocation point that triggered the ceiling
>> - *stop* mode, which stops the compilation similarly to how hitting the node limit in C2 would stop the compilation.
>> 
>> #### Usage
>> 
>> The compile command takes the form `-XX:CompileCommand:MemLimit,<method>,<memory size>[~<option>]`
>> 
>> The memory size that can be specified as usual. The command also takes an optional trailing option, separated from the memory size by a tilde (odd choice but needed because of https://bugs.openjdk.org/browse/JDK-8318214).
>> 
>> The `<suboption>` can be either `crash` or `stop` and distinguishes between the aforementioned modes. If omitted, the default is "stop".
>> 
>> Since `MemLimit` is based on the JIT statistic, it is implicitly enabled, so there is no need to explicitly enable it with `MemStat`.
>> 
>> 
>> Usage examples:
>> 
>> 
>> - crash when reaching 100m for a single compilation:
>>  
>> 
>> -XX:CompileCommand='MemLimit,*.*,100m~crash'
>> 
>> 
>> - limit all methods to a 2m footprint and print a trailing record at VM shutdown
>> 
>> 
>> -XX:CompileCommand='MemLimit,*.*,2m' -XX:CompileCommand='MemStat,*.*,print'
>> 
>> 
>> The JIT statistic table (printed either at VM shutdown or via `jcmd <pid> Compiler.memory`) now has a new column "result" that shows if the compilation was completed successfully or had an error or an oom:
>> 
>> 
>>                               vvvvv                                                                                                                                                                                                                                                
>> total     NA        RA        result  #nodes  time    type  #rc thread              method
>> 3036016   687328    1757456   oom     2607    0,320   c2    1   0x00007fcf1c181550  java/lang/invoke/StringConcatFactory::<clinit>(()V)
>> 
>> 
>> 
>> #### Implementation notes
>> 
>> The crash variant works immediately.
>> 
>> The stop variant works delayed: If arena memory reaches the limit in the scope of a compilation, we signal the Compile object (`Compile` fo...
>
> Thomas Stuefe has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains two additional commits since the last revision:
> 
>  - Merge branch 'openjdk:master' into JDK-8318016-compiler-memory-ceiling
>  - memlimit

That looks good to me.
Have you done any stress testing? Maybe running compile the world with a low memory limit.

src/hotspot/share/compiler/compilationMemoryStatistic.cpp line 400:

> 398: 
> 399:   // Store result
> 400:   // For this to work, we must call on_end_compilatio()n at a point where

Typo: on_end_compilatio()n

-------------

Marked as reviewed by roland (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/16220#pullrequestreview-1701208036
PR Review Comment: https://git.openjdk.org/jdk/pull/16220#discussion_r1374210686