Greetings,
I'm reaching out because of an issue with the JVM on MacOS that is hitting us a large scale. It started in MacOS 13 but got really worse in 14.1. We basically now need to use -XX:-MaxFDLimit on MacOS for everything with a classpath of more then 10k jars (monolith). That applies to the Java app itself as well as any Java based IDE (Eclipse, IntelliJ) or build tool.
The reason is that the MaxFDLimit implementation on Mac is broken in the JVM. The JVM is applying a lower limit to itself. We discovered the -XX:-MaxFDLimit solution after our old workarounds of increasing the open files on MacOS stopped working.
We discovered it in the Bazel repository:
https://github.com/bazelbuild/bazel/blob/699208763906fbd4b5e46e445b637436ee2293aa/src/main/cpp/startup_options.cc#L589-L597
// Disable the JVM's own unlimiting of file descriptors. We do this
// ourselves in blaze.cc so we want our setting to propagate to the JVM.
//
// The reason to do this is that the JVM's unlimiting is suboptimal on
// macOS. Under that platform, the JVM limits the open file descriptors
// to the OPEN_MAX constant... which is much lower than the per-process
// kernel allowed limit of kern.maxfilesperproc (which is what we set
// ourselves to).
In older versions of macOS we used a launch daemon to increase the limit:
❯ cat /Library/LaunchDaemons/limit.maxfiles.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>limit.maxfiles</string>
<key>ProgramArguments</key>
<array>
<string>/bin/bash</string>
<string>-c</string>
<string>launchctl limit maxfiles unlimited unlimited ; launchctl limit maxfiles 1000000 2147483647</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>ServiceIPC</key>
<false/>
</dict>
</plist>
This stopped working in MacOS 14.1
❯ sudo launchctl limit maxfiles 1000000 2147483647
Could not set resource limits: 150: Operation not permitted while System Integrity Protection is engaged
We got an interesting info from Apple:
Defaults
Soft - 65535
Hard - unlimited
Per Apple - The unlimited keyword is now an alias for 2147483647 (INT32_MAX) where before it was an alias for 10240 (OPEN_MAX).
With SIP enabled, the soft limit can be raised, but not lowered once it’s been raised. This is by design and will generate the "Could not set resource limits".
So perhaps the code in the JVM should be changed to INT32_MAX?
The thing is, we cannot add the flag uniquely. For example, on older Linux installations (eg., Ubuntu 20) we noticed the -XX:-MaxFDLimit is harmful, i.e. we actually do need the JVM to increase its file handles.
I found a couple old bugs but nothing actionable. Can this issue be revisited? I am not able to open a JDK bug, though.
Thanks a lot!
-Gunnar
--
Gunnar Wagenknecht
gunnar at wagenknecht.org, http://guw.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/hotspot-dev/attachments/20231213/e6dc1e76/attachment-0001.htm>