RFR: JDK-8318016: Per-compilation memory ceiling [v2]

[Thomas Stuefe]

> This RFE introduces a new compile command to limit memory usage per compilation. It complements and is based upon the JIT memory statistic introduced with https://bugs.openjdk.org/browse/JDK-8317683.
> 
> This new memory limit is applied during compilation for either C1 or C2. It has one of two modes:
> - *crash* mode (mainly for compiler devs), which aborts with hs-err file when the ceiling is reached, right in the compiler thread at the allocation point that triggered the ceiling
> - *stop* mode, which stops the compilation similarly to how hitting the node limit in C2 would stop the compilation.
> 
> #### Usage
> 
> The compile command takes the form `-XX:CompileCommand:MemLimit,<method>,<memory size>[~<option>]`
> 
> The memory size that can be specified as usual. The command also takes an optional trailing option, separated from the memory size by a tilde (odd choice but needed because of https://bugs.openjdk.org/browse/JDK-8318214).
> 
> The `<suboption>` can be either `crash` or `stop` and distinguishes between the aforementioned modes. If omitted, the default is "stop".
> 
> Since `MemLimit` is based on the JIT statistic, it is implicitly enabled, so there is no need to explicitly enable it with `MemStat`.
> 
> 
> Usage examples:
> 
> 
> - crash when reaching 100m for a single compilation:
>  
> 
> -XX:CompileCommand='MemLimit,*.*,100m~crash'
> 
> 
> - limit all methods to a 2m footprint and print a trailing record at VM shutdown
> 
> 
> -XX:CompileCommand='MemLimit,*.*,2m' -XX:CompileCommand='MemStat,*.*,print'
> 
> 
> The JIT statistic table (printed either at VM shutdown or via `jcmd <pid> Compiler.memory`) now has a new column "result" that shows if the compilation was completed successfully or had an error or an oom:
> 
> 
>                               vvvvv                                                                                                                                                                                                                                                
> total     NA        RA        result  #nodes  time    type  #rc thread              method
> 3036016   687328    1757456   oom     2607    0,320   c2    1   0x00007fcf1c181550  java/lang/invoke/StringConcatFactory::<clinit>(()V)
> 
> 
> 
> #### Implementation notes
> 
> The crash variant works immediately.
> 
> The stop variant works delayed: If arena memory reaches the limit in the scope of a compilation, we signal the Compile object (`Compile` for c2, `Compilation` for c1) that an oom has occurred.
> 
> For C1, we set the bailout string rig...

Thomas Stuefe has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains two additional commits since the last revision:

 - Merge branch 'openjdk:master' into JDK-8318016-compiler-memory-ceiling
 - memlimit

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/16220/files
  - new: https://git.openjdk.org/jdk/pull/16220/files/976d0114..8e0ab821

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=16220&range=01
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=16220&range=00-01

  Stats: 7394 lines in 213 files changed: 4202 ins; 2038 del; 1154 mod
  Patch: https://git.openjdk.org/jdk/pull/16220.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/16220/head:pull/16220

PR: https://git.openjdk.org/jdk/pull/16220