RFR: 8357135: java.lang.OutOfMemoryError: Error creating or attaching to libjvmci after JDK-8356447 [v2]

[Yudi Zheng]

On Tue, 20 May 2025 12:14:07 GMT, Doug Simon <dnsimon at openjdk.org> wrote:

>> As of [JDK-8356447](https://bugs.openjdk.org/browse/JDK-8356447), libgraal initialization happens during VM startup. If during this initialization, the libgraal heap cannot be created due to lack of virtual address space, the VM will exit with:
>> 
>> 
>> Error occurred during initialization of VM
>> java.lang.OutOfMemoryError: Error creating or attaching to libjvmci (err: -1000000801, description: Reserving address space for the new isolate failed.)
>> 
>> 
>> This causes problems for tests that limit the virtual address space with `ulimit -v` such as `gc/arguments/TestUseCompressedOopsFlagsWithUlimit.java` and `vmTestbase/nsk/jvmti/Allocate/alloc001/alloc001.java`.
>> Instead of exiting the VM, the failure should be silent (unless `-XX:+PrintCompilation` is enabled) as the VM can continue without libgraal, albeit in a crippled state. This PR implements this solution.
>> 
>> Alternative solutions include:
>> 1. Trying to adjust the values used with `ulimit -v` in the tests to accommodate the [virtual address reservations](https://github.com/oracle/graal/blob/69f10d3d658a6aeca3d5ce59c64af6a18336f14c/substratevm/src/com.oracle.svm.core.genscavenge/src/com/oracle/svm/core/genscavenge/AddressRangeCommittedMemoryProvider.java#L150) needed by libgraal. This is brittle as it assumes knowledge about how much address space is needed (which is turn depends on how many libgraal compiler threads are created).
>> 2. Add a `@requires !vm.libgraal.jit` guard to the tests so they are not run when libgraal is in use.
>> 
>> I think the solution in this PR is the most robust for the long term.
>
> Doug Simon has updated the pull request incrementally with one additional commit since the last revision:
> 
>   consolidate JVMCI eager initialization

LGTM

-------------

Marked as reviewed by yzheng (Committer).

PR Review: https://git.openjdk.org/jdk/pull/25307#pullrequestreview-2853970394