parallel IO anomaly with windows REFS formatted drive
Michael Skells
mike.skells1 at gmail.com
Sun Oct 29 22:09:23 UTC 2017
Hi
It seems to me that some NIO based file creates don't scale work well with
REFS formatted drives
REFS is an alternative (to NTFS) drive format for Windows 8 and above
When working with NTFS on my test system, as we use more threads the
performance improves so some degree, (1000 files creates take 5-600 ms with
one thread and 2-300 ms with 4 threads)
when using REFS the single threaded times are roughly the same for 1 and 2
threads, but get much worse for 3 and 4 threads (1200 - 1600 ms)
this showdown is only when using FileChanel.open and synchronous IO, for
the AsynchronousFileChannel it seems roughly the to scale as expected
CSV attached for the test results, and some test code. The test code is in
scala, and I can convert if needed, but it just really 3 calls, to open
write and close the files
My test system is an I7 4 core window 10 pro with Norton installed, but an
exclusion for the specified drives, and indexing disabled. All writes are
to an M.2 SSD with write-though cache
Some background
I am working on the back end code generation of the scala compiler which
generates many thousand small .class files, and part of that is to tune
the IO. For windows where the IO cost is significant, so I had a few test
app to simulate and measure the relative performance of the different
alternatives
Any ideas for a bulk file creation alternative welcome. I know we could
write to a JAR but that requires other re-tooling
Regards
Mike
More information about the core-libs-dev
mailing list