RFR: 8366298: FDLeakTest sometimes takes minutes to complete on Linux

[Roger Riggs]

On Thu, 28 Aug 2025 07:46:36 GMT, Stefan Karlsson <stefank at openjdk.org> wrote:

> While investigating [JDK-8260555](https://bugs.openjdk.org/browse/JDK-8260555), which lowers the timeout factor from 4 to 1, we found that FDLeakTest sometimes times out on Linux.
> 
> The reason is that the test performs a `fcntl` call for each and every potential file descriptor number. This can be a large number of calls and sometimes results in minutes-long test executions.
> 
> I propose that we fix this by limiting the max number of open file descriptors. This lowers the test execution time to about 1 second.
> 
> The test has two processes. One that executes the libFDLeaker.c code below as an agent in the test JVM, then it forks into a exeFDLeakTester.c, which reads the property `int max_fd = (int)sysconf(_SC_OPEN_MAX);`. The setting of `RLIMIT_NOFILE` to `100` lowers `max_fd` to `100`. I've verified this on both Linux and on macOS.
> 
> I've run the test manually on Linux and macOS and verified that it runs faster. I've also run this through tier1.

@tstuefe Please comment if you have concerns about this change.  Thanks

-------------

PR Comment: https://git.openjdk.org/jdk/pull/26979#issuecomment-3233559784