RFR: 8366118: DontCompileHugeMethods is not respected with -XX:-TieredCompilation

[Jiangli Zhou]

On Mon, 25 Aug 2025 19:38:23 GMT, Man Cao <manc at openjdk.org> wrote:

> Hi,
> 
> Could anyone review this change that fixes https://bugs.openjdk.org/browse/JDK-8366118? When this bug happens, it is difficult or almost impossible to debug due to the lack of stack trace, hs-err log or core dump. Fortunately we are also experimenting with sigaltstack for https://bugs.openjdk.org/browse/JDK-8364654, and it helped immensely to identify the root cause.
> 
> I will also try adding a test case for DontCompileHugeMethod under -XX:-TieredCompilation.
> 
> -Man

src/hotspot/share/compiler/compilationPolicy.cpp line 925:

> 923:   }
> 924: 
> 925:   if (!CompilationModeFlag::disable_intermediate()) {

AFAICT, the block of code here is intended for handling the case when intermediate is not disabled. Your change subtly alters that. 

When `TieredCompilation` is disabled, the large method compilation is done via  `CompileBroker::compile_method` if `!CompileBroker::compilation_is_in_queue(mh)` is `true`. I confirmed that in lldb, see below. Is there any reason to not do `can_be_compiled` check when calling `CompileBroker::compile_method`?

Additionally, should `can_be_compiled` check only be done for c2 compilation or if it should also be applied to c1 compilation? 


(lldb) bt
* thread #18, name = 'ApexEnumTest_de', stop reason = step in
  * frame #0: 0x00007ffff49b70ae libjvm.so`CompileBroker::compile_method(method=0x00007ffff38dba10, osr_bci=-1, comp_level=4, hot_method=0x00007ffff38dba10, hot_count=6784, compile_reason=Reason_Tiered, __the_thread__=0x00001354ff8c9810) at compileBroker.cpp:1347:21
    frame #1: 0x00007ffff49981fb libjvm.so`CompilationPolicy::compile(mh=0x00007ffff38dba10, bci=-1, level=CompLevel_full_optimization, __the_thread__=0x00001354ff8c9810) at compilationPolicy.cpp:824:5
    frame #2: 0x00007ffff4997baf libjvm.so`CompilationPolicy::method_invocation_event(mh=0x00007ffff38dba10, imh=0x00007ffff38dba10, level=CompLevel_none, nm=0x0000000000000000, __the_thread__=0x00001354ff8c9810) at compilationPolicy.cpp:1160:7
    frame #3: 0x00007ffff49979ea libjvm.so`CompilationPolicy::event(method=0x00007ffff38dba10, inlinee=0x00007ffff38dba10, branch_bci=-1, bci=-1, comp_level=CompLevel_none, nm=0x0000000000000000, __the_thread__=0x00001354ff8c9810) at compilationPolicy.cpp:745:5
    frame #4: 0x00007ffff4d79dd8 libjvm.so`InterpreterRuntime::frequency_counter_overflow_inner(current=0x00001354ff8c9810, branch_bcp=0x0000000000000000) at interpreterRuntime.cpp:1066:21
    frame #5: 0x00007ffff4d79a76 libjvm.so`InterpreterRuntime::frequency_counter_overflow(current=0x00001354ff8c9810, branch_bcp=0x0000000000000000) at interpreterRuntime.cpp:1015:17
    frame #6: 0x00007fffe1c0ce41
    frame #7: 0x00007fffe1c080a8
    frame #8: 0x00007fffe1c00d01
    frame #9: 0x00007ffff4d8786d libjvm.so`JavaCalls::call_helper(result=0x00007ffff38dc040, method=0x00007ffff38dbf90, args=0x00007ffff38dbec8, __the_thread__=0x00001354ff8c9810) at javaCalls.cpp:415:7
    frame #10: 0x00007ffff522bd74 libjvm.so`os::os_exception_wrapper(f=(libjvm.so`JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*) at javaCalls.cpp:332), value=0x00007ffff38dc040, method=0x00007ffff38dbf90, args=0x00007ffff38dbec8, thread=0x00001354ff8c9810) at os_linux.cpp:5190:3
    frame #11: 0x00007ffff4d8685a libjvm.so`JavaCalls::call(result=0x00007ffff38dc040, method=0x00007ffff38dbf90, args=0x00007ffff38dbec8, __the_thread__=0x00001354ff8c9810) at javaCalls.cpp:329:3
    frame #12: 0x00007ffff4e4a434 libjvm.so`jni_invoke_static(env=0x00001354ff8c9c10, result=0x00007ffff38dc040, receiver=0x0000000000000000, call_type=JNI_STATIC, method_id=0x00001354ffa77218, args=0x00007ffff38dc000, __the_thread__=0x00001354ff8c9810) at jni.cpp:888:3
    frame #13: 0x00007ffff4e4d815 libjvm.so`jni_CallStaticVoidMethodV(env=0x00001354ff8c9c10, cls=0x00001354ffdbd412, methodID=0x00001354ffa77218, args=0x00007ffff38dc180) at jni.cpp:1728:3
    frame #14: 0x00005555565a0bd0 ApexEnumTest_deploy.jar`JNIEnv_::CallStaticVoidMethod(_jclass*, _jmethodID*, ...) + 154
    frame #15: 0x00005555565a0064 ApexEnumTest_deploy.jar`devtools_java_launcher::internal::LauncherMainImpl::JavaMain(JNIEnv_*, std::__u::pair<std::__u::basic_string<char, std::__u::char_traits<char>, std::__u::allocator<char>>, devtools_java_launcher::internal::ArgumentEncoding> const&, int, char**, std::__u::basic_string<char, std::__u::char_traits<char>, std::__u::allocator<char>> const&) + 314
    frame #16: 0x00005555565a7e01 ApexEnumTest_deploy.jar`jvalue std::__u::__function::__policy_func<jvalue (JNIEnv_*)>::__call_func<std::__u::__bind_front_t<devtools_java_launcher::LauncherMain::RunJavaMainInJVMThread() const::$_0, std::__u::pair<std::__u::basic_string<char, std::__u::char_traits<char>, std::__u::allocator<char>>, devtools_java_launcher::internal::ArgumentEncoding>, int, char**, std::__u::basic_string<char, std::__u::char_traits<char>, std::__u::allocator<char>>>>(std::__u::__function::__policy_storage const*, JNIEnv_*) + 39
    frame #17: 0x00005555565b32af ApexEnumTest_deploy.jar`devtools_java_launcher::internal::UserRequest::operator()(thread::Future<devtools_java_launcher::JniResult>*) + 413
    frame #18: 0x00005555565b8e84 ApexEnumTest_deploy.jar`thread::Future<devtools_java_launcher::JniResult>::Impl::RunProducerAndUnref() + 102
    frame #19: 0x00005555565ba513 ApexEnumTest_deploy.jar`util::functional::internal::FunctorCallback<Closure, false, absl::AnyInvocable<void () &&>, void ()>::Run() + 89
    frame #20: 0x00005555565b4122 ApexEnumTest_deploy.jar`devtools_java_launcher::internal::LauncherImpl::VmThreadRoutine(void*) + 98
    frame #21: 0x00007ffff7e407db libpthread.so.0`start_thread + 187
    frame #22: 0x00007ffff7db305f libc.so.6`__clone + 63

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/26932#discussion_r2299117559