Memory Mapped Segment with offsets into the underlying file

[Maurizio Cimadamore]

Hi,

On 01/07/2020 11:04, Johannes Lichtenberger wrote:
> Hi,
>
> is it currently possible to specify a start offset somehow to map a
> specific region, despite the number of bytes to map? Im using Java 14 as of
> now.
the Java 15 API will let you do that - it takes both an offset and a 
length (in bytes).
>
> As already mentioned, my application is always appending data to a file and
> needs to read randomly. For writing I thought I could use a
> RandomAccessFile or Channel based implementation and only mmap the file for
> read-only operations. However, somehow, when I write a long offset into the
> position 0 and afterwards try to read it via:
>
> dataFileSegment =
>      MemorySegment.mapFromPath(checkNotNull(dataFile),
> dataFile.toFile().length(), FileChannel.MapMode.READ_ONLY);
>
> final MemoryAddress baseAddress = dataFileSegment.baseAddress();
>
> uberPageReference.setKey((long) LONG_VAR_HANDLE.get(baseAddress));
>
>
> the key I'm setting seems to be way off. However, it clearly might be
> a bug in my implementation or it's somehow not synchronized when
> something is written to the file without the Foreign Memory API or it
> might well be an anti-pattern.

Hard to say from here - while bugs in the API impl are always possible, 
I'd double check your impl first, since it seems like what you are doing 
is not trivial - in the sense that you end up with two views of the same 
file.

One thing to check is whether the writes using the regular IO API have 
been flushed to the file before the file is memory mapped.

Other than that your snippet above looks ok, and I don't think it should 
trigger any issue in the impl.

>
> Now I thought I could just map a region, which starts at the end of
> the current file for appending operations and spans maybe 1Gb for
> instance.
>
> Then I could save in a field the real length of the written data and
> adapt it everytime. When the mapped segment doesn't have enough space
> simply a new segment is created. Before closing the segment I'd have
> to truncate the size to the real length, however.
>
> Furthermore, I think it would be great if the JVM would have support for
> madvise to get rid of the prefetching of pages done by the Kernel in my
> case (random reads, as it's a tree of tries... but basically as in every
> index structure).

This was brought up before [1] - in general I think that, instead of 
adding XYZ feature to the JDK (most of which are going to be heavily 
OS-dependent), for advanced use cases it would probably best to just use 
`mmap` directly, as described in [2]. Of course that will only be 
possible once the FFI support is in, but we're not too far from that ;-)

Maurizio

[1] - 
https://mail.openjdk.java.net/pipermail/panama-dev/2020-April/008680.html
[2] - https://gist.github.com/mcimadamore/128ee904157bb6c729a10596e69edffd


>
> And as suggested I now try to map large regions of the file (or the whole
> file) plus setting VarHandles as static final fields :-)
>
> kind regards
> Johannes