JDK 10 RFR of 8184157: (ch) AsynchronousFileChannel hangs with internal error when reading locked file

[Brian Burkhalter]

https://bugs.openjdk.java.net/browse/JDK-8184157
http://cr.openjdk.java.net/~bpb/8184157/webrev.00/

This problem is caused by a race condition wherein the PendingFuture of a lock operation in the PendingIoCache is replaced by that of a read operation, with both using the same key, before the completion status of the lock operation is dequeued by the event handler thread of Iocp. When the lock completion status is dequeued, the event handler looks up its corresponding PendingFuture but retrieves that of the read operation instead. This causes the completed() callback of the read task to be invoked with the NumberOfBytes transferred for the lock operation which is a garbage value. This results in an attempt to set the read ByteBuffer position to a value outside its permitted range.

The proposed solution is to introduce an invalidate() method on the PendingIoCache which removes the entry corresponding to the OVERLAPPED pointer from the map of overlapped-to-PendingFuture entries and places the OVERLAPPED pointer value in a Set where it remains until remove() is called. This ensures that the OVERLAPPED pointer map key will not be used for a PendingFuture to which it does not correspond. The pointer values of invalidated PendingFutures are removed from the Set when remove() is called in the Iocp event handler after the corresponding completion status is dequeued. All existing regression tests pass with this patch applied.

Thanks,

Brian