RFR: 8371944: AOT configuration is corrupted when app closes System.out

[David Holmes]

On Sat, 15 Nov 2025 05:34:32 GMT, Vladimir Kozlov <kvn at openjdk.org> wrote:

>> During an AOT training run, some application may close `System.out` (usually inadvertently by naive code like the following):
>> 
>> 
>> try (var err = new PrintWriter(System.err);
>>       var out = new PrintWriter(System.out)) {
>>      out.println("Hello Confused World");
>> }
>> 
>> 
>> This has the side effect of closing the JVM's STDOUT.
>> 
>> When the JVM is about to exit, we will open the AOT configuration file for writing. On Windows we may get back a file HANDLE (which is just an integer) that's identical to the now closed STDOUT.
>> 
>> If the JVM writes to STDOUT (usually with UL logging), it will corrupt the contents of the AOT configuration file.
>> 
>> This doesn't happen on Posix as `System.out.close()` will keep file descriptions 1 and 2 open, preventing them from being reused by files that are opened in the future.
>> 
>> The fix is to open the AOT configuration file early, before any application code is executed. That way we can guarantee that we will have a file HANDLE (or file descriptor) that's different than STDOUT.
>> 
>> The test failed failed about 20% of the time on Windows before the fix. After the fix, it ran 100 times without failure.
>
> The fix seems fine for this case.
> 
> VM itself may open a lot of files for different logs (JFR, JIT logs, UL, etc). May VM close some before you open AOT file which could cause the same issue? Is it possible to track life time of such files to make sure we don't write into them after they are closed?

@vnkozlov I think stdout/stderr are special in this regard because the Java code closed a handle that the VM is still actively using. It would be a clear bug in the VM if we closed a file whilst still being used by the VM code.

That said this still seems like a VM bug to me. If Java code closes System.out/err that should not affect stdout/stderr as used by the VM. Maybe the fact this isn't an issue on Posix masked the fact it is a general issue. We should be able to dup the Handle in the VM so that it can't possibly later refer to a different output device.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/28335#issuecomment-3544795786