RFR: 8351733: [macos] Crash when creating too many nested event loops

[Jay Bhaskar]

On Fri, 21 Mar 2025 20:56:01 GMT, Martin Fox <mfox at openjdk.org> wrote:

> There is an undocumented limit on nesting calls to CFRunLoopRun (or the equivalent wrapper NSRunLoop methods). When the limit is hit the OS terminates the Java app. The situation arises when a JavaFX app creates too many nested event loops from within  Platform.runLater runnables.
> 
> This PR doesn't change the limit (which is 250+ nested loops) but it does throw an exception just before the limit is reached so a JavaFX developer will get a useful Java stack trace instead of an OS crash log.
> 
> On the Mac the nested event loop has two stages: first we ask the run loop to run, then we pull an event out and process it. A Platform.runLater runnable is executed in the first stage so if the runnable starts a new nested event loop the system will re-enter CFRunLoopRun. The same isn't true if an input event handler starts a new nested event loop; at that point we're in stage two and are past the call to CFRunLoopRun.

It is better to use dynamically calculated safe limit instead of a hardcoded value like 250. The dynamic calculation should be based on system resource and and 250 would be fallback value. like 
static int getSafeRunLoopLimit() {
    struct rlimit limit;
    if (getrlimit(RLIMIT_STACK, &limit) == 0) {
        return (int)(limit.rlim_cur / (512 * 1024)); 
    }
    return 250; // Fallback to 250 if stack size retrieval fails
}
macOS does not document an exact limit, nested CFRunLoopRun calls can cause stack growth. 512 KB is a safe approximation.

-------------

PR Comment: https://git.openjdk.org/jfx/pull/1741#issuecomment-2744896944