RFR: 8376125: Out of memory in the CDS archive error with lot of classes [v3]

[Ashutosh Mehra]

On Tue, 3 Feb 2026 04:10:41 GMT, Ioi Lam <iklam at openjdk.org> wrote:

>> I was on MacOS.  Here is the failure without the os.cpp update:
>> 
>> % make test TEST="test/hotspot/jtreg/resourcehogs/runtime/aot/LargeArchive.java" JTREG="JAVA_OPTIONS=-Dtest.archive.large.all.workflows=true"
>> ...
>> [0.018s][info][aot] JVM_StartThread() ignored: java.lang.ref.Reference$ReferenceHandler
>> [0.018s][info][aot] JVM_StartThread() ignored: java.lang.ref.Finalizer$FinalizerThread
>> [0.039s][info][cds] Loading classes to share ...
>> [0.039s][info][cds] Parsing LargeArchive.classlist
>> [0.047s][info][aot] JVM_StartThread() ignored: jdk.internal.misc.InnocuousThread
>> [136.124s][info][cds] Parsing /Users/xuelei.fan/workspace/openjdk/jdk-xf.git/build/macosx-aarch64-server-release/images/jdk/lib/classlist (lambda form invokers only)
>> [136.126s][info][cds] Loading classes to share: done.
>> [136.127s][info][aot] Rewriting and linking classes ...
>> [136.245s][info][aot] Rewriting and linking classes: done
>> [136.245s][info][aot] Regenerate MethodHandle Holder classes...
>> [136.344s][info][aot] Regenerate MethodHandle Holder classes...done
>> [136.351s][info][cds] Dumping shared data to file: LargeArchive.static.jsa
>> [136.351s][info][cds] Gathering all archivable objects ... 
>> [136.371s][info][cds] Skipping java/lang/invoke/BoundMethodHandle$Species_F: dynamically generated
>> [136.371s][info][cds] Skipping java/lang/invoke/BoundMethodHandle$Species_J: dynamically generated
>> [136.413s][info][cds] Skipping jdk/internal/misc/CDS$UnregisteredClassLoader$Source: used only when dumping CDS archive
>> [136.413s][info][cds] Skipping jdk/internal/misc/CDS$UnregisteredClassLoader: used only when dumping CDS archive
>> [136.417s][info][cds] Heap range = [0x00000003c0000000 - 0x00000004c0000000]
>> [136.430s][info][aot] Archived 7975 interned strings
>> [136.431s][info][cds] Gathering classes and symbols ... 
>> [143.110s][info][cds] Sorting symbols ... 
>> [143.113s][info][cds] Sorting classes ... 
>> [149.873s][info][cds] Reserved output buffer space at 0x0000007000000000 [34359738368 bytes]
>> [149.900s][info][cds] Allocating RW objects ... 
>> [149.954s][info][cds] done (46510 objects)
>> [149.954s][info][cds] Allocating RO objects ... 
>> [151.033s][info][cds] done (142334 objects)
>> [151.033s][info][cds] Relocating embedded pointers in core regions ... 
>> [152.400s][info][cds] Relocating 150345176 pointers, 0 tagged, 17461 nulled
>> [152.400s][info][aot] Make classes shareable
>> [152.558s][info][cds] Number of classes 4485
>> [152.558s][info][cds]     instance class...
>
> According to https://gitlab.haskell.org/ghc/ghc/-/issues/17414 
> 
>> File reads/writes bigger than 2GB result in an "Invalid argument" exception on macOS. Files bigger than 2GB still work, but individual read/write operations bigger than 2GB fail.
> 
> I think it's better to move this fix into `os::pd_write()` (within `#ifdef __APPLE__`) to limit the writes to less than 2GB.

@iklam thanks for digging that up. It explains why the INT_MAX limit worked. But I should also mention that the above output does not show the actual reason for the failure. Here the `os::write` failed causing the `fd` to be closed in `FileMapInfo::write_bytes`. However, the error is not propagated up the call chain and we end up calling `FileMapInfo::seek_to_position` which throws EBADF (Bad file descriptor).
So while we can keep the change in `os::write` (or `os::pd_write` as suggested) I think we should also fix `FileMapInfo::write_bytes` to 1) print the os error, and 2) terminate the write operation gracefully. I am also fine if this is done in a follow-up pr.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/29494#discussion_r2757134702