RFR: 8318302: ThreadCountLimit.java failed with "Native memory allocation (mprotect) failed to protect 16384 bytes for memory to guard stack pages"

[Aleksey Shipilev]

On Thu, 22 Feb 2024 07:36:15 GMT, David Holmes <dholmes at openjdk.org> wrote:

> The test tries to create as many threads as possible in 5 seconds, with the intention of hitting platform limits on the maximum number of threads that can be created, so that we see that the VM reacts in a reasonable way. However on Linux, if the thread creation limit is high enough, and things run fast enough, then we can instead hit a fatal error when a mmap/mprotect returns E_NOMEM because the maximum number of memory mappings (default 65530) has been exceeded.
> 
> The fix to the test (courtesy @stefank ) is to re-exec the test on Linux with `ulimit -u` set to a much smaller value than the default (I chose 4096 as it still showed over 2000 threads were created before failure). This means we are unlikely to hit the default memory mapping limit before we hit the max threads limit.
> 
> I also updated the VM's OOM error message to include exceeding the maximum number of memory mappings as a possible cause (with a hint as to what to look for in the hs_err log).
> 
> Testing:
>  - ran the test 10x on all platforms in our CI (plus local testing)
> 
> Thanks

Looks fine to me, with nits.

Maybe deserves a little synopsis change: "8318302: ThreadCountLimit.java failed due to too many memory mappings"

test/hotspot/jtreg/runtime/Thread/ThreadCountLimit.java line 80:

> 78:         final String ULIMIT_CMD = "ulimit -u 4096";
> 79:         ProcessBuilder pb = ProcessTools.createTestJavaProcessBuilder(ThreadCountLimit.class.getName());
> 80:         String java_cmd = ProcessTools.getCommandLine(pb);

Nit: `java_cmd` -> `javaCmd`.

test/hotspot/jtreg/runtime/Thread/ThreadCountLimit.java line 82:

> 80:         String java_cmd = ProcessTools.getCommandLine(pb);
> 81:         // Relaunch the test with args.length > 0, and the ulimit set
> 82:         ProcessTools.executeCommand("bash", "-c", ULIMIT_CMD + " && " + java_cmd + " dummy")

I think this assumes `bash` is installed, which is true 99% of the time, but not actually guaranteed. Maybe we should check `if (Platform.isLinux() && new File("/bin/bash").exists())` explicitly.

test/hotspot/jtreg/runtime/Thread/ThreadCountLimit.java line 142:

> 140:     // Now that all threads have joined, we are away from dangerous
> 141:     // VM state and have enough memory to perform any other things.
> 142:     if (!reachedNativeOOM) {

Cleanup: flip this around to avoid negation in `!reachedNativeOOM`.

-------------

Marked as reviewed by shade (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/17959#pullrequestreview-1895288969
PR Comment: https://git.openjdk.org/jdk/pull/17959#issuecomment-1959016262
PR Review Comment: https://git.openjdk.org/jdk/pull/17959#discussion_r1498901287
PR Review Comment: https://git.openjdk.org/jdk/pull/17959#discussion_r1498907529
PR Review Comment: https://git.openjdk.org/jdk/pull/17959#discussion_r1498895112