8194734 : Handle to jimage file inherited into child processes (win)

[Alexander Miloslavskiy]

The problem in this bug is that jimage file is mistakenly opened with 
"inherit by child processes" flag. In our case, the child process is 
started with Runtime.exec() and serves as updater that can also replace 
embedded JRE. However, due to jimage ("lib/modules") file handle being 
inherited, embedded JRE can not be replaced.

The bug was introduced in commits "8087181: Move native jimage code to 
its own library (maybe libjimage)"

Before this commit, os::open() was used in libjimage code and it handles 
file inheritance correctly, see:
/hotspot/src/share/vm/classfile/imageFile.cpp
* ImageFileReader::open() uses os::open(_name, 0, O_RDONLY)
/hotspot/src/os/windows/vm/os_windows.cpp
* os::open() calls to ::open(pathbuf, oflag | O_BINARY | O_NOINHERIT, mode);
/hotspot/src/os/linux/vm/os_linux.cpp
* os::open() uses both O_CLOEXEC and FD_CLOEXEC
After this commit, a new osSupport::openReadOnly() is implemented and it 
does not handle file inheritance properly.

Please find patch attached.
As far as I understand from OpenJDK's "How to contribute" page, I need a 
"sponsor" who will get this patch applied.

Windows - compiled and tested:
Bug fixed.

Ubuntu  - compiled and tested:
Bug did not occur before my patch due to another bug in childProcess() 
function (src/java.base/unix/native/libjava/childproc.c) According to my 
debugging, FD_CLOEXEC is applied to jimage file there because 
childProcess() thinks that it's a error pipe (FAIL_FILENO). Should the 
bug in childProcess() be fixed, it can unearth currently fixed bug.
-------------- next part --------------
# HG changeset patch
# User Alexander Miloslavskiy <alexandr.miloslavskiy at gmail.com>
# Date 1522703926 -10800
#      Tue Apr 03 00:18:46 2018 +0300
# Node ID 87954e967cc67cab4b480a4ec7ff54a334e0f4ce
# Parent  de0fd2c8a401a564f06d42fac15559154c42358a
8194734: jimage file descriptor was inherited into processes started with Runtime.exec()
Summary: Prevented inheritance with FD_CLOEXEC on unix, O_NOINHERIT on windows.
Refactoring: added O_RDONLY (equals 0)
Refactoring: added O_BINARY on windows (currently ignored by osSupport::read and osSupport::map_memory)

diff -r de0fd2c8a401 -r 87954e967cc6 src/java.base/unix/native/libjimage/osSupport_unix.cpp
--- a/src/java.base/unix/native/libjimage/osSupport_unix.cpp	Fri Mar 30 14:36:18 2018 -0700
+++ b/src/java.base/unix/native/libjimage/osSupport_unix.cpp	Tue Apr 03 00:18:46 2018 +0300
@@ -38,7 +38,16 @@
  * Return the file descriptor.
  */
 jint osSupport::openReadOnly(const char *path) {
-    return ::open(path, 0);
+    int fd = ::open(path, O_RDONLY);
+
+    // jimage file descriptors must not be inherited by child processes.
+    // This is especially true for child processes started with Runtime.exec (see 8194734)
+    #ifdef FD_CLOEXEC
+        int flags = ::fcntl(fd, F_GETFD);
+        ::fcntl(fd, F_SETFD, flags | FD_CLOEXEC);
+    #endif
+
+    return fd;
 }
 
 /**
diff -r de0fd2c8a401 -r 87954e967cc6 src/java.base/windows/native/libjimage/osSupport_windows.cpp
--- a/src/java.base/windows/native/libjimage/osSupport_windows.cpp	Fri Mar 30 14:36:18 2018 -0700
+++ b/src/java.base/windows/native/libjimage/osSupport_windows.cpp	Tue Apr 03 00:18:46 2018 +0300
@@ -38,7 +38,9 @@
  * Return the file descriptor.
  */
 jint osSupport::openReadOnly(const char *path) {
-    return ::open(path, 0, 0);
+    // jimage file descriptors must not be inherited by child processes.
+    // This is especially true for child processes started with Runtime.exec (see 8194734)
+    return ::open(path, O_BINARY | O_NOINHERIT, O_RDONLY);
 }
 
 /**