RFR [15] 8236825: Reading output from ... using ProcessBuilder/Process might hang

[Roger Riggs]

Please review a potential way to address two issues of java.lang.Process 
deadlocks during process exit. [1] [2]
(Linxu OS process expertise appreciated).

The deadlock is between some thread reading process output from a 
process that has exited
and the processExited thread that is attempting to buffer any remaining 
output so
the file descriptor for the pipe can be closed.  The methods involved 
are synchronized
on a ProcessPipeInputStream instance.

The hard case arises infrequently since the pipe streams are closed by 
the OS
normally (or within a few seconds) and the readXXX completes.
However, the case causing trouble is when the subprocess has spawned
another process and both processes are using the same file 
descriptor/stream for output.
Though the process that exits doesn't have the fd open anymore the extra 
subprocess does.
And if that subprocess does not exit, then the read and deadlock does 
not get resolved.

The approach proposed is to use a semaphore to guard the readXXX and
providing some non-blocking logic in processExited to forcibly close
the pipe if it detects that there is a conflicting read in progress.
(And remove the synchronized on processExited).

This solution works ok on MacOSX, where one of the issues occurred 
frequently.
Closing the pipe unblocks the reading thread.

On Linux, it appears that the blocking read (in native code) does not 
unblock
unless a signal occurs; so the solution does not fix the problem 
adqurated/completely.

Having a non-blocking native read would be the next step of complexity.
The problem has been around for a while so it may be an option to wait
for additional work on non-blocking pipe reads, the direction that Loom 
is moving.

Suggestions welcome,

Thanks, Roger

Webrev: http://cr.openjdk.java.net/~rriggs/webrev-hdiutil-8236825/

Issues:
[1] https://bugs.openjdk.java.net/browse/JDK-8236825
[2] https://bugs.openjdk.java.net/browse/JDK-8169565