RFR: 8366118: DontCompileHugeMethods is not respected with -XX:-TieredCompilation

[Man Cao]

On Mon, 25 Aug 2025 20:54:27 GMT, Jiangli Zhou <jiangli at openjdk.org> wrote:

> AFAICT, the block of code here is intended for handling the case when intermediate is not disabled. Your change subtly alters that.
> When TieredCompilation is disabled, the large method compilation is done via CompileBroker::compile_method if !CompileBroker::compilation_is_in_queue(mh) is true. I confirmed that in lldb, see below. Is there any reason to not do can_be_compiled check when calling CompileBroker::compile_method?

Trying to compile the large method under `-XX:-TieredCompilation` is the bug. The large method should not be compiled under `-XX:+DontCompileHugeMethods`.

The bug is caused by erroneously guarding the `!can_be_compiled()` and `!can_be_osr_compiled()` checks behind `!CompilationModeFlag::disable_intermediate()`. The correct behavior is to do the following checks and returns regardless of `TieredCompilation`:

if ((bci == InvocationEntryBci && !can_be_compiled(mh, level))) {
  return;
}
if ((bci != InvocationEntryBci && !can_be_osr_compiled(mh, level))) {
  return;
}

Only the recursive call to `compile(mh, bci, CompLevel_simple, THREAD)` and `osr_nm->make_not_entrant()` need to be guarded under `!disable_intermediate()`.

It is possible to add the above two checks for `bci`, `can_be_compiled()` and `!can_be_osr_compiled()` to inside `CompileBroker::compile_method()`, specifically inside `CompileBroker::compilation_is_prohibited()`. If compiler-dev team prefers this way, we could move them.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/26932#discussion_r2299288827