RFR: 8366298: FDLeakTest sometimes takes minutes to complete on Linux [v2]

[Thomas Stuefe]

On Fri, 29 Aug 2025 09:00:20 GMT, Stefan Karlsson <stefank at openjdk.org> wrote:

>> While investigating [JDK-8260555](https://bugs.openjdk.org/browse/JDK-8260555), which lowers the timeout factor from 4 to 1, we found that FDLeakTest sometimes times out on Linux.
>> 
>> The reason is that the test performs a `fcntl` call for each and every potential file descriptor number. This can be a large number of calls and sometimes results in minutes-long test executions.
>> 
>> I propose that we fix this by limiting the max number of open file descriptors. This lowers the test execution time to about 1 second.
>> 
>> The test has two processes. One that executes the libFDLeaker.c code below as an agent in the test JVM, then it forks into a exeFDLeakTester.c, which reads the property `int max_fd = (int)sysconf(_SC_OPEN_MAX);`. The setting of `RLIMIT_NOFILE` to `100` lowers `max_fd` to `100`. I've verified this on both Linux and on macOS.
>> 
>> I've run the test manually on Linux and macOS and verified that it runs faster. I've also run this through tier1.
>
> Stefan Karlsson has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Add error logging

Good, thanks!

-------------

Marked as reviewed by stuefe (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/26979#pullrequestreview-3168592578