Memory Mapped Segment with offsets into the underlying file

[Johannes Lichtenberger]

I think I'll wait until Java 15 is shipped, as I also need concurrent
access sometimes (for instance when I'm auto-committing data with a
ScheduledExecutorService).

Kind regards
Johannes




Am Mi., 1. Juli 2020 um 13:43 Uhr schrieb Maurizio Cimadamore <
maurizio.cimadamore at oracle.com>:

> Hi,
>
> On 01/07/2020 11:04, Johannes Lichtenberger wrote:
> > Hi,
> >
> > is it currently possible to specify a start offset somehow to map a
> > specific region, despite the number of bytes to map? Im using Java 14 as
> of
> > now.
> the Java 15 API will let you do that - it takes both an offset and a
> length (in bytes).
> >
> > As already mentioned, my application is always appending data to a file
> and
> > needs to read randomly. For writing I thought I could use a
> > RandomAccessFile or Channel based implementation and only mmap the file
> for
> > read-only operations. However, somehow, when I write a long offset into
> the
> > position 0 and afterwards try to read it via:
> >
> > dataFileSegment =
> >      MemorySegment.mapFromPath(checkNotNull(dataFile),
> > dataFile.toFile().length(), FileChannel.MapMode.READ_ONLY);
> >
> > final MemoryAddress baseAddress = dataFileSegment.baseAddress();
> >
> > uberPageReference.setKey((long) LONG_VAR_HANDLE.get(baseAddress));
> >
> >
> > the key I'm setting seems to be way off. However, it clearly might be
> > a bug in my implementation or it's somehow not synchronized when
> > something is written to the file without the Foreign Memory API or it
> > might well be an anti-pattern.
>
> Hard to say from here - while bugs in the API impl are always possible,
> I'd double check your impl first, since it seems like what you are doing
> is not trivial - in the sense that you end up with two views of the same
> file.
>
> One thing to check is whether the writes using the regular IO API have
> been flushed to the file before the file is memory mapped.
>
> Other than that your snippet above looks ok, and I don't think it should
> trigger any issue in the impl.
>
> >
> > Now I thought I could just map a region, which starts at the end of
> > the current file for appending operations and spans maybe 1Gb for
> > instance.
> >
> > Then I could save in a field the real length of the written data and
> > adapt it everytime. When the mapped segment doesn't have enough space
> > simply a new segment is created. Before closing the segment I'd have
> > to truncate the size to the real length, however.
> >
> > Furthermore, I think it would be great if the JVM would have support for
> > madvise to get rid of the prefetching of pages done by the Kernel in my
> > case (random reads, as it's a tree of tries... but basically as in every
> > index structure).
>
> This was brought up before [1] - in general I think that, instead of
> adding XYZ feature to the JDK (most of which are going to be heavily
> OS-dependent), for advanced use cases it would probably best to just use
> `mmap` directly, as described in [2]. Of course that will only be
> possible once the FFI support is in, but we're not too far from that ;-)
>
> Maurizio
>
> [1] -
> https://mail.openjdk.java.net/pipermail/panama-dev/2020-April/008680.html
> [2] - https://gist.github.com/mcimadamore/128ee904157bb6c729a10596e69edffd
>
>
> >
> > And as suggested I now try to map large regions of the file (or the whole
> > file) plus setting VarHandles as static final fields :-)
> >
> > kind regards
> > Johannes
>