RFR: 8371944: AOT configuration is corrupted when app closes System.out

[Ioi Lam]

On Sat, 15 Nov 2025 05:34:32 GMT, Vladimir Kozlov <kvn at openjdk.org> wrote:

>> During an AOT training run, some application may close `System.out` (usually inadvertently by naive code like the following):
>> 
>> 
>> try (var err = new PrintWriter(System.err);
>>       var out = new PrintWriter(System.out)) {
>>      out.println("Hello Confused World");
>> }
>> 
>> 
>> This has the side effect of closing the JVM's STDOUT.
>> 
>> When the JVM is about to exit, we will open the AOT configuration file for writing. On Windows we may get back a file HANDLE (which is just an integer) that's identical to the now closed STDOUT.
>> 
>> If the JVM writes to STDOUT (usually with UL logging), it will corrupt the contents of the AOT configuration file.
>> 
>> This doesn't happen on Posix as `System.out.close()` will keep file descriptions 1 and 2 open, preventing them from being reused by files that are opened in the future.
>> 
>> The fix is to open the AOT configuration file early, before any application code is executed. That way we can guarantee that we will have a file HANDLE (or file descriptor) that's different than STDOUT.
>> 
>> The test failed failed about 20% of the time on Windows before the fix. After the fix, it ran 100 times without failure.
>
> The fix seems fine for this case.
> 
> VM itself may open a lot of files for different logs (JFR, JIT logs, UL, etc). May VM close some before you open AOT file which could cause the same issue? Is it possible to track life time of such files to make sure we don't write into them after they are closed?

@vnkozlov 

Only file descriptors 0, 1, and 2 can be closed by user apps. E.g.,`FileDescriptor.out`:

https://github.com/openjdk/jdk/blob/696821670e11fee003906806f081038032ac4985/src/java.base/share/classes/java/io/FileDescriptor.java#L159

Other files opened by the JVM (JFR, etc) cannot be closed by the app, as there are public no APIs to get their file descriptors.

On Unix, file descriptors 0, 1, and 2 are treated specially in io_util_md.c. We keep those file descriptors valid using `dup2()`, so they can never be recycled to files that are opened in the future:

https://github.com/openjdk/jdk/blob/696821670e11fee003906806f081038032ac4985/src/java.base/unix/native/libjava/io_util_md.c#L161-L168

However, there are no such handling on Windows:

https://github.com/openjdk/jdk/blob/696821670e11fee003906806f081038032ac4985/src/java.base/windows/native/libjava/io_util_md.c#L551-L553

On Windows, each open file is represented by a `HANDLE`, which is a 32- or 64-bit integer. After `System.out.close()` is called from the app, the `HANDLE` that represented STDOUT is closed, so `tty->print()` will stop printing to the stdout. However, sometimes (the integer value of) this `HANDLE` can be reused by Windows when opening a new file, thus causing `tty->print()` to write to the new file, leading to the current issue.

The `dup2()` call on Unix basically prevents the file descriptor to be reused. As far as I know, there are no equivalent APIs on Windows.

@dholmes-ora 

> If Java code closes System.out/err that should not affect stdout/stderr as used by the VM. Maybe the fact this isn't an issue on Posix masked the fact it is a general issue. We should be able to dup the Handle in the VM so that it can't possibly later refer to a different output device.

This would change the behavior of the past 25 years, where `System.out.close()` has meant "this current process should no longer write to the stdout out". This applies to Java code, JVM, and JNI code.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/28335#issuecomment-3544883037