RFR: 8331185: Enable compiler memory limits in debug builds

[Vladimir Kozlov]

On Fri, 26 Apr 2024 10:04:28 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

> See [1] for previous discussions.
> 
> We'd like to introduce a default memory limit for compilations in debug builds. That way, we can catch pathological compiler errors that have an unreasonably high per-compilation memory footprint early during testing.
> 
> The default limit affects all compilations, unless the method is subject to a memory limit set from command line. Meaning, `-XX:CompileCommand=MemLimit,...` overrules the default.
> 
> Examples:
> 
> This lowers the memlimit for j.l.String methods - all methods will have the default 1GB limit in a debug JVM. Only j.l.String will run with a 100M limit: `-XX:CompileCommand=MemLimit,java.lang.String::*,100m`
> 
> This disables the default memlimit globally: `-XX:CompileCommand=MemLimit,*.*,0`
> 
> 
> ---
> 
> The patch:
> 
> 1) adds a debug-only default memory limit of **1GB** (as proposed by @vnkozlov). The limit action is "crash", meaning we will assert.
> 2) To test the mechanics, we now print out the memory limit for each compilation in the compilation cost record.
> 3) Adapted and extended tests
> 
> I also fixed up some copyrights that I overlooked last year when adding the compiler memory statistics this patch builds atop of.
> 
> 
> Tested:
> 
> - manually on Mac m1 (debug and release)
> - GHAs are running
> - but Oracle will do more testing before this goes in
> 
> [1] https://mail.openjdk.org/pipermail/hotspot-compiler-dev/2024-April/074787.html

`compiler/c2/TestFindNode.java` and `compiler/loopopts/TestDeepGraphVerifyIterativeGVN.java` failed on `aarch64` when run with stress flags:

`-XX:+UnlockDiagnosticVMOptions -XX:-TieredCompilation -XX:+StressArrayCopyMacroNode -XX:+StressLCM -XX:+StressGCM -XX:+StressIGVN -XX:+StressCCP -XX:+StressMacroExpansion -XX:+StressMethodHandleLinkerInlining -XX:+StressCompiledExceptionHandlers`


#  Internal Error (/workspace/open/src/hotspot/share/compiler/compilationMemoryStatistic.cpp:551), pid=912622, tid=912639
#  fatal error: c2 compiler/c2/TestFindNode::test(()V): Hit MemLimit (limit: 1073741824 now: 1073824544)

V  [libjvm.so+0x97496c]  report_fatal(VMErrorType, char const*, int, char const*, ...)+0x108  (debug.cpp:214)
V  [libjvm.so+0x8a9e2c]  CompilationMemoryStatistic::on_arena_change(long, Arena const*)+0x5bc  (compilationMemoryStatistic.cpp:551)
V  [libjvm.so+0x58ee70]  Arena::grow(unsigned long, AllocFailStrategy::AllocFailEnum)+0x10c  (arena.cpp:300)
V  [libjvm.so+0x13eb86c]  PhaseChaitin::Split(unsigned int, ResourceArea*)+0x3ac  (resourceArea.inline.hpp:35)
V  [libjvm.so+0x798a44]  PhaseChaitin::Register_Allocate()+0x554  (chaitin.cpp:553)
V  [libjvm.so+0x8ced88]  Compile::Code_Gen()+0x284  (compile.cpp:2988)
V  [libjvm.so+0x8d1128]  Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)+0x1588  (compile.cpp:896)
V  [libjvm.so+0x72d480]  C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)+0x17c  (c2compiler.cpp:142)


Looks like Register Allocator eats memory.

There also few closed tests which failed with `-XX:-TieredCompilation -XX:+AlwaysIncrementalInline` on `x64` and `aarch64`:


V  [libjvm.so+0xa90b95]  report_fatal(VMErrorType, char const*, int, char const*, ...)+0x105  (debug.cpp:214)
V  [libjvm.so+0x9c06dd]  CompilationMemoryStatistic::on_arena_change(long, Arena const*)+0x5cd  (compilationMemoryStatistic.cpp:551)
V  [libjvm.so+0x5f45fc]  Arena::grow(unsigned long, AllocFailStrategy::AllocFailEnum)+0x10c  (arena.cpp:300)
V  [libjvm.so+0x14e4706]  Parse::init_blocks()+0x46  (parse1.cpp:1290)
V  [libjvm.so+0x14f0aff]  Parse::Parse(JVMState*, ciMethod*, float)+0x52f  (parse1.cpp:565)
V  [libjvm.so+0x8412b9]  ParseGenerator::generate(JVMState*)+0x169  (callGenerator.cpp:99)
V  [libjvm.so+0x84482a]  PredictedCallGenerator::generate(JVMState*)+0x3aa  (callGenerator.cpp:928)
V  [libjvm.so+0x8466de]  CallGenerator::do_late_inline_helper()+0x90e  (callGenerator.cpp:704)
V  [libjvm.so+0x9e2de4]  Compile::inline_incrementally_one()+0xd4  (compile.cpp:2054)
V  [libjvm.so+0x9e3b32]  Compile::inline_incrementally(PhaseIterGVN&)+0x292  (compile.cpp:2137)
V  [libjvm.so+0x9e5b50]  Compile::Optimize()+0x340  (compile.cpp:2272)
V  [libjvm.so+0x9e9c60]  Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)+0x1b50  (compile.cpp:863)
V  [libjvm.so+0x83ed15]  C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)+0x1d5  (c2compiler.cpp:142)

-------------

PR Comment: https://git.openjdk.org/jdk/pull/18969#issuecomment-2080024827
PR Comment: https://git.openjdk.org/jdk/pull/18969#issuecomment-2080030280