RFR: 8308094: Add a compilation timeout flag to catch long running compilations [v8]

[Christian Hagedorn]

On Thu, 14 Aug 2025 14:57:36 GMT, Manuel Hässig <mhaessig at openjdk.org> wrote:

>> This PR adds `-XX:CompileTaskTimeout` on Linux to limit the amount of time a compilation task can run. The goal of this is initially to be able to find and investigate long-running compilations.
>> 
>> The timeout is implemented using a POSIX timer that sends a `SIGALRM` to the compiler thread the compile task is running on. Each compiler thread registers a signal handler that triggers an assert upon receiving `SIGALRM`. This is currently only implemented for Linux, because it relies on `SIGEV_THREAD_ID` to get the signal delivered to the same thread that timed out.
>> 
>> Since `SIGALRM` is now used, the test `runtime/signal/TestSigalrm.java` now requires `vm.flagless` so it will not interfere with the compiler thread signal handlers.
>> 
>> Testing:
>>  - [x] Github Actions
>>  - [x] tier1, tier2 on all platforms
>>  - [x] tier3, tier4 and Oracle internal testing on Linux fastdebug
>>  - [x] tier1 through tier4 with `-XX:CompileTaskTimeout=60000` (one minute timeout) to see what fails (`compiler/codegen/TestAntiDependenciesHighMemUsage2.java`, `compiler/loopopts/TestMaxLoopOptsCountReached.java`, and `compiler/c2/TestScalarReplacementMaxLiveNodes.java` fail)
>
> Manuel Hässig has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Fix format string

Thanks a lot for adding a test and printing the method name! A few more comments but otherwise, it looks good now!

src/hotspot/os/linux/compilerThreadTimeout_linux.cpp line 44:

> 42:       CompileTask* task = CompilerThread::current()->task();
> 43:       assert(false, "compile task %d (%s) timed out after " INTPTR_FORMAT " ms",
> 44:              task->compile_id(), task->method()->name_and_sig_as_C_string(), CompileTaskTimeout);

Normally, you would probably need a `ResourceMark` for getting the method name. However, I'm not sure if you can do that as well here inside the signal handler. Maybe someone else can comment on that.

test/hotspot/jtreg/compiler/arguments/TestCompileTaskTimeout.java line 29:

> 27:  * @test TestCompileTaskTimeout
> 28:  * @bug 8308094
> 29:  * @requires vm.compiler2.enabled & vm.debug & vm.flagless & os.name == "Linux"

Does it only work with C2 compile tasks?

test/hotspot/jtreg/compiler/arguments/TestCompileTaskTimeout.java line 40:

> 38: 
> 39:     public static void main(String[] args) throws Throwable {
> 40:         ProcessTools.executeTestJava("-Xcomp", "-XX:CompileTaskTimeout=1", "-version")

Nit: I think for newer JDK versions after JDK 8, we should use `--version`.

-------------

Marked as reviewed by chagedorn (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/26023#pullrequestreview-3123183236
PR Review Comment: https://git.openjdk.org/jdk/pull/26023#discussion_r2278487336
PR Review Comment: https://git.openjdk.org/jdk/pull/26023#discussion_r2278479526
PR Review Comment: https://git.openjdk.org/jdk/pull/26023#discussion_r2278490157