RFR: 8308094: Add a compilation timeout flag to catch long running compilations [v4]

[Dean Long]

On Thu, 7 Aug 2025 14:52:48 GMT, Manuel Hässig <mhaessig at openjdk.org> wrote:

>> This PR adds `-XX:CompileTaskTimeout` on Linux to limit the amount of time a compilation task can run. The goal of this is initially to be able to find and investigate long-running compilations.
>> 
>> The timeout is implemented using a POSIX timer that sends a `SIGALRM` to the compiler thread the compile task is running on. Each compiler thread registers a signal handler that triggers an assert upon receiving `SIGALRM`. This is currently only implemented for Linux, because it relies on `SIGEV_THREAD_ID` to get the signal delivered to the same thread that timed out.
>> 
>> Since `SIGALRM` is now used, the test `runtime/signal/TestSigalrm.java` now requires `vm.flagless` so it will not interfere with the compiler thread signal handlers.
>> 
>> Testing:
>>  - [ ] Github Actions
>>  - [ ] tier1, tier2 on all platforms
>>  - [ ] tier3, tier4 and Oracle internal testing on Linux fastdebug
>>  - [ ] tier1 through tier4 with `-XX:CompileTaskTimeout=60000` (one minute timeout) to see what fails (`compiler/codegen/TestAntiDependenciesHighMemUsage2.java`, `compiler/loopopts/TestMaxLoopOptsCountReached.java`, and `compiler/c2/TestScalarReplacementMaxLiveNodes.java` fail)
>
> Manuel Hässig has updated the pull request incrementally with one additional commit since the last revision:
> 
>   ASSERT

Thinking about _timeout_armed a little more, the fact the the signal handler received TIMEOUT_SIGNAL should be enough.  The value of _timeout_armed should be redundant, and your assert could be changed to:

assert(false, "compile task timed out");

and _timeout_armed could be removed. It's just an inexact mirror of the timer state.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/26023#issuecomment-3165377812