RFR: 8366298: FDLeakTest sometimes takes minutes to complete on Linux

[Thomas Stuefe]

On Thu, 28 Aug 2025 07:46:36 GMT, Stefan Karlsson <stefank at openjdk.org> wrote:

> While investigating [JDK-8260555](https://bugs.openjdk.org/browse/JDK-8260555), which lowers the timeout factor from 4 to 1, we found that FDLeakTest sometimes times out on Linux.
> 
> The reason is that the test performs a `fcntl` call for each and every potential file descriptor number. This can be a large number of calls and sometimes results in minutes-long test executions.
> 
> I propose that we fix this by limiting the max number of open file descriptors. This lowers the test execution time to about 1 second.
> 
> The test has two processes. One that executes the libFDLeaker.c code below as an agent in the test JVM, then it forks into a exeFDLeakTester.c, which reads the property `int max_fd = (int)sysconf(_SC_OPEN_MAX);`. The setting of `RLIMIT_NOFILE` to `100` lowers `max_fd` to `100`. I've verified this on both Linux and on macOS.
> 
> I've run the test manually on Linux and macOS and verified that it runs faster. I've also run this through tier1.

Thanks for looking at this. See comments inline.

test/jdk/java/lang/ProcessBuilder/FDLeakTest/libFDLeaker.c line 54:

> 52:   if (ret != 0) {
> 53:     return JNI_ERR;
> 54:   }

Hmm, if the JVM already has opened > 100 fds at this point, we get difficult to understand errors. Any future open etc call will fail.

What I would do is to check, after the call to fopen() at line 39, check for errno==EMFILE. If that is true, write out a clear error message along the lines of "more than 100 files open?" (since that in itself would be worth to investigate; this little test should not open > 100 files, but who knows).

-------------

PR Review: https://git.openjdk.org/jdk/pull/26979#pullrequestreview-3167841516
PR Review Comment: https://git.openjdk.org/jdk/pull/26979#discussion_r2309518202