8194734 : Handle to jimage file inherited into child processes (win)

[Martin Buchholz]

On Mon, Apr 2, 2018 at 2:50 PM, Alexander Miloslavskiy <
alexandr.miloslavskiy at gmail.com> wrote:

> The problem in this bug is that jimage file is mistakenly opened with
> "inherit by child processes" flag. In our case, the child process is
> started with Runtime.exec() and serves as updater that can also replace
> embedded JRE. However, due to jimage ("lib/modules") file handle being
> inherited, embedded JRE can not be replaced.
>
> The bug was introduced in commits "8087181: Move native jimage code to its
> own library (maybe libjimage)"
>
> Before this commit, os::open() was used in libjimage code and it handles
> file inheritance correctly, see:
> /hotspot/src/share/vm/classfile/imageFile.cpp
> * ImageFileReader::open() uses os::open(_name, 0, O_RDONLY)
> /hotspot/src/os/windows/vm/os_windows.cpp
> * os::open() calls to ::open(pathbuf, oflag | O_BINARY | O_NOINHERIT,
> mode);
> /hotspot/src/os/linux/vm/os_linux.cpp
> * os::open() uses both O_CLOEXEC and FD_CLOEXEC
> After this commit, a new osSupport::openReadOnly() is implemented and it
> does not handle file inheritance properly.
>

I'm still only half paying attention, but I agree with Alexander that the
low-level infrastructure in libjimage is fishy.  The rule should be that
file descriptors created by the JDK should have the CLOEXEC bit on.

(Too many parts of the JDK create their own low-level C infrastructure -
we're not good at sharing.)

But even if the CLOEXEC bit was forgotten, the subprocess code should still
succeed in closing that fd and use FAIL_FILENO for the error reporting pipe
in childproc.c.