Memory Mapped Segment with offsets into the underlying file

[Maurizio Cimadamore]

I believe (as with other things mmap related) you are in OS territory. I 
think the upper limit there is just the virtual addressable space.

Maurizio

On 01/07/2020 19:03, Johannes Lichtenberger wrote:
> Another thing I couldn't get from reading the JavaDocs, what's the 
> biggest size I can map on 64 Bit systems?
>
> johannes at johannesl3:/opt$ uname -a
> Linux johannesl3 5.3.0-61-generic #55-Ubuntu SMP Fri Jun 19 11:16:34 
> UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
>
> I tried
>
> MemorySegment.mapFromPath(dataFile, Long.MAX_VALUE, 
> FileChannel.MapMode.READ_WRITE);
> and some other values. Integer.MAX_VALUE is permitted at least :-)
> kind regards
> Johannes
>
> Am Mi., 1. Juli 2020 um 13:43 Uhr schrieb Maurizio Cimadamore 
> <maurizio.cimadamore at oracle.com <mailto:maurizio.cimadamore at oracle.com>>:
>
>     Hi,
>
>     On 01/07/2020 11:04, Johannes Lichtenberger wrote:
>     > Hi,
>     >
>     > is it currently possible to specify a start offset somehow to map a
>     > specific region, despite the number of bytes to map? Im using
>     Java 14 as of
>     > now.
>     the Java 15 API will let you do that - it takes both an offset and a
>     length (in bytes).
>     >
>     > As already mentioned, my application is always appending data to
>     a file and
>     > needs to read randomly. For writing I thought I could use a
>     > RandomAccessFile or Channel based implementation and only mmap
>     the file for
>     > read-only operations. However, somehow, when I write a long
>     offset into the
>     > position 0 and afterwards try to read it via:
>     >
>     > dataFileSegment =
>     >      MemorySegment.mapFromPath(checkNotNull(dataFile),
>     > dataFile.toFile().length(), FileChannel.MapMode.READ_ONLY);
>     >
>     > final MemoryAddress baseAddress = dataFileSegment.baseAddress();
>     >
>     > uberPageReference.setKey((long) LONG_VAR_HANDLE.get(baseAddress));
>     >
>     >
>     > the key I'm setting seems to be way off. However, it clearly
>     might be
>     > a bug in my implementation or it's somehow not synchronized when
>     > something is written to the file without the Foreign Memory API
>     or it
>     > might well be an anti-pattern.
>
>     Hard to say from here - while bugs in the API impl are always
>     possible,
>     I'd double check your impl first, since it seems like what you are
>     doing
>     is not trivial - in the sense that you end up with two views of
>     the same
>     file.
>
>     One thing to check is whether the writes using the regular IO API
>     have
>     been flushed to the file before the file is memory mapped.
>
>     Other than that your snippet above looks ok, and I don't think it
>     should
>     trigger any issue in the impl.
>
>     >
>     > Now I thought I could just map a region, which starts at the end of
>     > the current file for appending operations and spans maybe 1Gb for
>     > instance.
>     >
>     > Then I could save in a field the real length of the written data and
>     > adapt it everytime. When the mapped segment doesn't have enough
>     space
>     > simply a new segment is created. Before closing the segment I'd have
>     > to truncate the size to the real length, however.
>     >
>     > Furthermore, I think it would be great if the JVM would have
>     support for
>     > madvise to get rid of the prefetching of pages done by the
>     Kernel in my
>     > case (random reads, as it's a tree of tries... but basically as
>     in every
>     > index structure).
>
>     This was brought up before [1] - in general I think that, instead of
>     adding XYZ feature to the JDK (most of which are going to be heavily
>     OS-dependent), for advanced use cases it would probably best to
>     just use
>     `mmap` directly, as described in [2]. Of course that will only be
>     possible once the FFI support is in, but we're not too far from
>     that ;-)
>
>     Maurizio
>
>     [1] -
>     https://mail.openjdk.java.net/pipermail/panama-dev/2020-April/008680.html
>     [2] -
>     https://gist.github.com/mcimadamore/128ee904157bb6c729a10596e69edffd
>
>
>     >
>     > And as suggested I now try to map large regions of the file (or
>     the whole
>     > file) plus setting VarHandles as static final fields :-)
>     >
>     > kind regards
>     > Johannes
>