RFR: JDK-8318016: Per-compilation memory ceiling [v3]

[Thomas Stuefe]

> This RFE introduces a new compile command to limit memory usage per compilation. It complements and is based upon the JIT memory statistic introduced with https://bugs.openjdk.org/browse/JDK-8317683.
> 
> This new memory limit is applied during compilation for either C1 or C2. It has one of two modes:
> - *crash* mode (mainly for compiler devs), which aborts with hs-err file when the ceiling is reached, right in the compiler thread at the allocation point that triggered the ceiling
> - *stop* mode, which stops the compilation similarly to how hitting the node limit in C2 would stop the compilation.
> 
> #### Usage
> 
> The compile command takes the form `-XX:CompileCommand:MemLimit,<method>,<memory size>[~<option>]`
> 
> The memory size that can be specified as usual. The command also takes an optional trailing option, separated from the memory size by a tilde (odd choice but needed because of https://bugs.openjdk.org/browse/JDK-8318214).
> 
> The `<suboption>` can be either `crash` or `stop` and distinguishes between the aforementioned modes. If omitted, the default is "stop".
> 
> Since `MemLimit` is based on the JIT statistic, it is implicitly enabled, so there is no need to explicitly enable it with `MemStat`.
> 
> 
> Usage examples:
> 
> 
> - crash when reaching 100m for a single compilation:
>  
> 
> -XX:CompileCommand='MemLimit,*.*,100m~crash'
> 
> 
> - limit all methods to a 2m footprint and print a trailing record at VM shutdown
> 
> 
> -XX:CompileCommand='MemLimit,*.*,2m' -XX:CompileCommand='MemStat,*.*,print'
> 
> 
> The JIT statistic table (printed either at VM shutdown or via `jcmd <pid> Compiler.memory`) now has a new column "result" that shows if the compilation was completed successfully or had an error or an oom:
> 
> 
>                               vvvvv                                                                                                                                                                                                                                                
> total     NA        RA        result  #nodes  time    type  #rc thread              method
> 3036016   687328    1757456   oom     2607    0,320   c2    1   0x00007fcf1c181550  java/lang/invoke/StringConcatFactory::<clinit>(()V)
> 
> 
> 
> #### Implementation notes
> 
> The crash variant works immediately.
> 
> The stop variant works delayed: If arena memory reaches the limit in the scope of a compilation, we signal the Compile object (`Compile` for c2, `Compilation` for c1) that an oom has occurred.
> 
> For C1, we set the bailout string rig...

Thomas Stuefe has updated the pull request incrementally with one additional commit since the last revision:

  fix typo

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/16220/files
  - new: https://git.openjdk.org/jdk/pull/16220/files/8e0ab821..55e12f68

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=16220&range=02
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=16220&range=01-02

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jdk/pull/16220.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/16220/head:pull/16220

PR: https://git.openjdk.org/jdk/pull/16220