RFR: JDK-8318016: Per-compilation memory ceiling

[Thomas Stuefe]

This RFE introduces a new compile command to limit memory usage per compilation. It complements and is based upon the JIT memory statistic introduced with https://bugs.openjdk.org/browse/JDK-8317683.

This new memory limit is applied during compilation for either C1 or C2. It has one of two modes:
- *crash* mode (mainly for compiler devs), which aborts with hs-err file when the ceiling is reached, right in the compiler thread at the allocation point that triggered the ceiling
- *stop* mode, which stops the compilation similarly to how hitting the node limit in C2 would stop the compilation.

#### Usage

The compile command takes the form `-XX:CompileCommand:MemLimit,<method>,<memory size>[~<option>]`

The memory size that can be specified as usual. The command also takes an optional trailing option, separated from the memory size by a tilde (odd choice but needed because of https://bugs.openjdk.org/browse/JDK-8318214).

The `<suboption>` can be either `crash` or `stop` and distinguishes between the aforementioned modes. If omitted, the default is "stop".

Since `MemLimit` is based on the JIT statistic, it is implicitly enabled, so there is no need to explicitly enable it with `MemStat`.


Usage examples:


- crash when reaching 100m for a single compilation:
 

-XX:CompileCommand='MemLimit,*.*,100m~crash'


- limit all methods to a 2m footprint and print a trailing record at VM shutdown


-XX:CompileCommand='MemLimit,*.*,2m' -XX:CompileCommand='MemStat,*.*,print'


The JIT statistic table (printed either at VM shutdown or via `jcmd <pid> Compiler.memory`) now has a new column "result" that shows if the compilation was completed successfully or had an error or an oom:


                              vvvvv                                                                                                                                                                                                                                                
total     NA        RA        result  #nodes  time    type  #rc thread              method
3036016   687328    1757456   oom     2607    0,320   c2    1   0x00007fcf1c181550  java/lang/invoke/StringConcatFactory::<clinit>(()V)



#### Implementation notes

The crash variant works immediately.

The stop variant works delayed: If arena memory reaches the limit in the scope of a compilation, we signal the Compile object (`Compile` for c2, `Compilation` for c1) that an oom has occurred.

For C1, we set the bailout string right here. That causes the C1 compilation to fail at the next bailout check.

For C2, we cannot just set `Compile::_failure_reason` since I found that the C2 is not well prepared to bail out from wherever it is tested. So I incorporated the oom check in the nodelimit check C2 does. Note that even that needs some preparatory work (see https://github.com/openjdk/jdk/pull/16205, a prerequisite for this RFE).


Due to this delay, the actual memory used per compilation may be somewhat higher than the limit. In practice the limit works reasonably well, though.

-------------

Commit messages:
 - memlimit

Changes: https://git.openjdk.org/jdk/pull/16220/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=16220&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8318016
  Stats: 429 lines in 17 files changed: 395 ins; 3 del; 31 mod
  Patch: https://git.openjdk.org/jdk/pull/16220.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/16220/head:pull/16220

PR: https://git.openjdk.org/jdk/pull/16220