Review request: JDK-7147084 (process) appA hangs when read output stream of appB which starts appC that runs forever

[Alexey Utkin]

Bug description:
https://jbs.oracle.com/bugs/browse/JDK-7147084
http://bugs.sun.com/view_bug.do?bug_id=7147084

Here is the suggested fix:
http://cr.openjdk.java.net/~uta/openjdk-webrevs/JDK-7147084/webrev.00/

Summary:
I would like to describe the bug synopsis, common Java approach to 
Windows handle inheritance and introduce the solution. The description 
covers the bugs JDK-7147084 
<https://jbs.oracle.com/bugs/browse/JDK-7147084>, JDK-6921885 
<https://jbs.oracle.com/bugs/browse/JDK-6921885>, JDK-6428742 
<https://jbs.oracle.com/bugs/browse/JDK-6428742>. All of them have the 
same source.

Inherited Windows handles are limited by IO and synchronization objects 
(read more at [1]). The handle itself is an integer key in a system hash 
map. The system hash map is unique for a process.

The parent process could share the handles with installed 
[HANDLE_FLAG_INHERIT] bit (the inherit-bit for short) with child process 
by setting the parameter [bInheritHandles] to [TRUE] in the Win32 API 
[CreateProcess] call [2]. Only the [CreateProcess] call is used by Java 
for new process creation. The term "share the handle" means that the 
same integer key (handle) is valid in both system hash maps: in parent 
and child processes.

Java does not provide the API to change the inherit-bit for any handle. 
More other, since at least the JDK 6, all Java-created handles have no 
installed inherit-bit .
Handles that change the inherit-bit to 1 in the Java call are the 
handles of redirected Input, Output, and Error streams (IOE streams for 
short) for child process. That is the only way to redirect the streams. 
That's why we can not give up the nomination in [TRUE] the parameter 
[bInheritHandles] in the [CreateProcess] call. And I want to mention 
again that this is the only place in JDK where Java installs the 
inherit-bit. Java itself does not use handle inheritance.

Below are the facts that are essential for further discussion, but not 
mentioned explicitly in the MS documentation:
- the only way to redirect the IOE streams of child process is to make 
they inherited in parent process and nominate in [TRUE] the parameter 
[bInheritHandles] in the [CreateProcess] call;
- handles of IOE streams in the child process have the same value and 
conserve the inherit-bit in 1;
- nomination of the inherit-bit bit to 0 in parent process does not drop 
the the inherit-bit in child process. As a result the grandchild process 
inherits the child process IOE handles;
- Java does not provide the API to change the inherit-bit, Java child 
process cannot drop the inherit-bit; child process can not suggest, 
should it drop the inherit-bit for IOE handles or not;
- each inheritance increase the handle reference counter;
- synchronous read/write operations for pipes cannot finish until 
another-end handle will not finally closed in all processes.

As a result of facts below we got two problems:
- if IOE handles made "inherit" in parent process, they conserve the 
inherit-bit in 1 at least in time of the [CreateProcess] call (in case 
we like to return the bit in previous value, that is not true for 
current implementation). That is rather long time. If the parallel 
thread will be fast enough, it will have time to make a competitive 
challenge to the [ProcessImpl.create] function and the competitive child 
process will inherit not belong to him handles. That is the subject of 
JDK-6921885 <https://jbs.oracle.com/bugs/browse/JDK-6921885> bug.
- the fact that in the child process handles of IOE streams have the 
same value and conserve the inherit-bit in 1 leads to "greeady 
grandchild" problem. That is a subject of JDK-7147084 
<https://jbs.oracle.com/bugs/browse/JDK-7147084>.

The common root of problems described in MS knowlage DB [3].

Suggested fix based on following facts:
- Java does not need the installed inherit-bit, except when starting a 
child process and only for the IOE handles
- (consequence of the first) In the child process IOE handles are the 
only that could get installed inherit-bit due to Java activity. Java is 
not interested in inheritance of IOE handle to grandchild process by 
default, but only as explicit IOE handle for grandchild.
- in accordance with MS kb315939 the synchronized [ProcessImpl.create] 
call is mandatory for JDK-6921885 
<https://jbs.oracle.com/bugs/browse/JDK-6921885> fix.
- any IOE standard handle should have the same value of inherit-bit 
before and after the [ProcessImpl.create] call to minimize the fix 
impact handle propagation in child processes.

References:
1. Handle Inheritance: 
http://msdn.microsoft.com/en-us/library/windows/desktop/ms724466%28v=vs.85%29.aspx 

2. CreateProcess function: 
http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425%28v=vs.85%29.aspx 

3. kb315939: http://support.microsoft.com/kb/315939

Regards,
-uta