RFR: 8371944: AOT configuration is corrupted when app closes System.out

[Ioi Lam]

On Sat, 15 Nov 2025 04:50:01 GMT, Ioi Lam <iklam at openjdk.org> wrote:

> During an AOT training run, some application may close `System.out` (usually inadvertently by naive code like the following):
> 
> 
> try (var err = new PrintWriter(System.err);
>       var out = new PrintWriter(System.out)) {
>      out.println("Hello Confused World");
> }
> 
> 
> This has the side effect of closing the JVM's STDOUT.
> 
> When the JVM is about to exit, we will open the AOT configuration file for writing. On Windows we may get back a file HANDLE (which is just an integer) that's identical to the now closed STDOUT.
> 
> If the JVM writes to STDOUT (usually with UL logging), it will corrupt the contents of the AOT configuration file.
> 
> This doesn't happen on Posix as `System.out.close()` will keep file descriptions 1 and 2 open, preventing them from being reused by files that are opened in the future.
> 
> The fix is to open the AOT configuration file early, before any application code is executed. That way we can guarantee that we will have a file HANDLE (or file descriptor) that's different than STDOUT.
> 
> The test failed failed about 20% of the time on Windows before the fix. After the fix, it ran 100 times without failure.

We could try to intercept the native calls to prevent writes to stdout/stderr when the underlying file descriptors have been closed by the Java program. However, we have very extensive use of C library functions such as `printf`, `fprinf`, `vfprinf`, etc, not just in HotSpot but in various other JDK libraries. So we would need to add an indirection in all those call sites.

Also, there's no way for us to intercept such calls from JNI libraries.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/28335#issuecomment-3544898959