RFR: 8357135: java.lang.OutOfMemoryError: Error creating or attaching to libjvmci after JDK-8356447 [v3]

[Doug Simon]

On Wed, 21 May 2025 20:41:35 GMT, Doug Simon <dnsimon at openjdk.org> wrote:

>> As of [JDK-8356447](https://bugs.openjdk.org/browse/JDK-8356447), libgraal initialization happens during VM startup. If during this initialization, the libgraal heap cannot be created due to lack of virtual address space, the VM will exit with:
>> 
>> 
>> Error occurred during initialization of VM
>> java.lang.OutOfMemoryError: Error creating or attaching to libjvmci (err: -1000000801, description: Reserving address space for the new isolate failed.)
>> 
>> 
>> This causes problems for tests that limit the virtual address space with `ulimit -v` such as `gc/arguments/TestUseCompressedOopsFlagsWithUlimit.java` and `vmTestbase/nsk/jvmti/Allocate/alloc001/alloc001.java`.
>> Since these tests were passing on libgraal prior to JDK-8356447, they obviously do not require JIT compilation. The simplest fix is to then use `-Xint` to disable the JIT.
>
> Doug Simon has updated the pull request incrementally with three additional commits since the last revision:
> 
>  - tests that use 'ulimit -v' should run with -Xint
>  - Revert "do not exit VM if libjvmci env creation fails"
>    
>    This reverts commit 7eb259b92553669065db57d230476cf465a67d02.
>  - Revert "consolidate JVMCI eager initialization"
>    
>    This reverts commit 32986d1a2b741ee8c9090cefbecc148bb8fbd7e4.

Tested locally with a build that includes libgraal.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/25307#issuecomment-2899184608