FIFO signaling producer that consumer has drained all bytes in pipe

[Jason Mehrens]

Hi Alan,

You are correct in that I was using JDK8.  Here is a sample output from JDK 11:
====
/usr/lib/jvm/java-11-openjdk-11.0.8.10-0.el7_8.x86_64
/tmp/FifoTest10748204695125312444.fifo
java.io.File::length 0
java.nio.file.Files::size 0
java.io.FileInputStream::available 312
RandomAccessFile at sun.nio.ch.FileChannelImpl::size 0
RandomAccessFile at java.io.FileInputStream::available 0
java.io.RandomAccessFile::length 0
sun.nio.ch.FileChannelImpl::size 0
====
***Given cat is started before writting, FIS::available results will vary as it does in this run***

So for #3 that is fixed if I stop using JDK 8.

If I understand correctly, it is intentional that FileChannel::size and RandomAccessFile::length return zero because tooling would use fstat64 too?  By tooling you mean external programs like 'ls'?  Essentially Files::size == FileChannel::size == RandomAccessFile::length == File::length.  Re-reading the FileChannel API it says, "Returns the current size of this channel's file."  So that seem like the correct implementation.  Per SeekableByteChannel, "Returns the current size of entity to which this channel is connected."  That seems like it should be doing whatever FIS::available is doing.  There isn't much utility in simply returning zero for size when read will return bytes from the channel.

The idea of checking for the pipe being empty is referenced in Dot Net in the following example as "pipeServer.WaitForPipeDrain();":
https://docs.microsoft.com/en-us/dotnet/standard/io/how-to-use-anonymous-pipes-for-local-interprocess-communication
I've yet to find the suggested implementation from that example though. :)


One of the constraints is that I can't modify the consumer program.  The actual consumer program reads an input file on start and is long running.  The ideal goal is to limit the lifetime of the input file.  Current implementation is creating actual temp files.  Then started exploring FIFOs, which evolved in demo program.  From what I'm gathering it seems I should abandon the idea of using RandomAccessFile/FileChannel/FileInputStream with FIFO.




Another thing I noticed is that FC::force fails against a FIFO:
====
Exception in thread "main" java.io.IOException: Invalid argument
	at java.base/sun.nio.ch.FileDispatcherImpl.force0(Native Method)
	at java.base/sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:82)
	at java.base/sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:461)
	at FifoTest.main(FifoTest.java:56)
======


Jason
________________________________________
From: Alan Bateman <Alan.Bateman at oracle.com>
Sent: Sunday, August 30, 2020 9:33 AM
To: Jason Mehrens; nio-dev at openjdk.java.net
Subject: Re: FIFO signaling producer that consumer has drained all bytes in pipe

On 29/08/2020 00:50, Jason Mehrens wrote:
> Hello nio-dev,
>
> Working on a program where a Java NIO producer of bytes is sending data through a FIFO (created with 'mkfifo' command) to an external process which is the consumer.
> The producer needs a signal that the consumer has completed consuming the bytes so it can be determined that the FIFO can be deleted by the producer.  I've included test case below to demonstrate the concept using cat as the consumer.  So far I think the test demonstrates viability but it has raised a few questions and it would be awesome to get some insights.
>
> 1. FileInputStream::available is the only method that seems to determine the number of bytes in the FIFO.  Is opening with SYNC redundant or necessary to ensure all parties communicate state correctly?
> 2. FileChannel::size always returns zero.  It would be much nicer if FileChannel::size returned bytes available.  Seems like returning zero violates SeekableByteChannel  and FileChannel specs.
> 3. RandomAccessFile::length fails with java.io.IOException: Illegal seek.  Spec seems to indicate that length should return length of the file.  Files(Path)::size, File.length and 'ls' all return zero.  The spec doesn't say anything on being able to seek to determine the length.
> 4. Is there any planned JDK support for FIFOs?  Currently, there is no way to create a FIFO from Java without resorting to calling an external process, JNI, etc.
> 5. Any insights on the demo program? What am I missing?
JEP 380 is adding support for Unix domain sockets and maybe that is an
option for what you are doing. The only support for pipes is the Pipe
API which where both ends are in the same VM. It's a bit of stretch to
try to use the FileChannel or the java.io APIs for FIFO special files
because many of the operations don't make sense for special files. I
can't make sense of some of your results, e.g. RAF::length uses fstat64,
not lseek64 but maybe you are on JDK 8? It should be feasible to have
FileChannel::size return the number of buffered bytes but that would
likely introduce an inconsistency with tooling that will return its size
of 0. On your overall "design", have you considered using two FIFOs so
that the consumer can communicate that is done?

-Alan