Recent Java 9 commit (e5b66323ae45) breaks fsync on directory

[Uwe Schindler]

Hi Alan,

Thank you for the quick response!

The requirement to fsync on the directory from Java came already quite often on the web (also before Java 7 release - but, before Java 7 it was impossible to do from Java code).

This is one example from before Java 7:
http://www.quora.com/Is-there-any-way-to-fsync%28%29-a-directory-from-Java

Stackoverflow has some questions about this, too. A famous one (ranked #1 at Google is this one):
http://stackoverflow.com/questions/7694307/using-filechannel-to-fsync-a-directory-with-nio-2

In fact this is exactly what we do in Lucene. The question here "Can I count on this working on all Unix platforms, in future versions of Java, and in non-Oracle JVMs?" is now answered -> NO.

Fsyncing on a directory is in most cases not needed for "standard java programs", but for those who really want to do this (like Lucene or Hadoop), maybe the idea with a separate OpenOption would be an idea! In Lucene code we can (which has Java 7 as minimum requirement) look with reflection for the new OpenOption and pass it. Unfortunately, people using currently released Lucene/Solr/Elasticsearch versions can no longer be sure that their index survives power outages, if they run it with Java 9. If we can early step in and test the new API, we can already release artifacts which at least "try" to use the new OpenOption (if available) and fall back to Java 7/Java 8 semantics otherwise.

Personally, I would prefer to just document that opening a file channel for sure only works with regular files, but may fail with other types of files (think of directories, or /dev/xxx devices). The code like it is now was working fine for 2 major Java releases, so why change semantics? If somebody accidentially opens a directory for reading, it is perfectly fine if he gets an IOException a bit delayed. If one opens a block device and writes a non block aligned bunch of data, it will fail, too. You patch does not handle this case, it only tests for directories. So I think we should leave it up to the operating system what you can do with a "file".

About Windows: In fact, you can also open a directory with CreateFile() [1], but with standard flags this fails with access denied (this is what we see in Java 7 and Java 8). You have to pass FILE_FLAG_BACKUP_SEMANTICS as flags, then you can do it [2]. But FlushFileBuffers does not work, because [2] does not list it as "valid call" for directory file handles (because its not needed for Windows, in opposite to POSIX). So FileChannel#force() would still fail. In that case. For Linux it is important that opening directories only works with READ, but not with WRITE, but this is obvious.

You may also want to read Mike McCandless blog about testing this with his installation using remote power switches on a testing machine to test durability. With the current Java 7 code in Lucene he got no failures:
http://blog.mikemccandless.com/2014/04/testing-lucenes-index-durability-after.html 

Uwe

[1] http://msdn.microsoft.com/en-us/library/windows/desktop/aa363858(v=vs.85).aspx
[2] http://msdn.microsoft.com/en-us/library/windows/desktop/aa365258(v=vs.85).aspx

> > We really would like to keep the possibility to fsync a directory on
> supported operating systems. We hope that the above commit will not be
> backported into 8u40 and 7u80 releases! In Java 9 we can discuss about other
> solutions how to handle this:
> > - Keep current semantics as of Java 7 and Java 8 and just fail if you really
> want to READ/WRITE from this FileChannel? This is how the underlying
> operatinmg system and libc handles this. You can open a file descriptor on
> anything, file/directory/device/..., but not all operations work on this
> descriptor, some of them throw exception/return error.
> > - Add a new API for fsyncing a directory (maybe for any file type). Like
> Files.fsync(Path)? On Windows this could just be a no-op for directories?
> Basically something like our IOUtils.fsync() from the link above.
> >
> > What's you opinion and how should we proceed?
> >
> This use-case may need a new API, one possibility is a new OpenOption (like
> NOFOLLOW_LINKS) for opening directories. This would allow opening a
> FileChannel to a directory and also provides somewhere to specify that many
> of the operations may fail. Implementation-wise it also means you should be
> able to open directories on Windows.
> 
> Sorry the original change broken what you are doing but I'm sure you
> understand that the unspecified behavior to allow directories be opened on
> some platforms and have subsequent attempts to do common operations
> (like read) fail wasn't ideal either. There are no plans to back-port this
> change.
> 
> -Alan