Memory Mapped file segment (file is empty)
Maurizio Cimadamore
maurizio.cimadamore at oracle.com
Thu Jun 25 14:03:59 UTC 2020
Support for NVM has been added as part of this JEP:
https://openjdk.java.net/jeps/352
Note that all this JEP does is, really, to provide two additional map
modes (and their corresponding implementation in the JDK of course,
which is where all the work is).
I believe that you should be able to use these map modes to create a
mapped segment which under the hood uses the MAP_SYNC mmap mode (where
supported).
> When a persistent-memory region is mapped using MAP_SYNC, the
> memory-management code will check to see whether there are metadata
> writes pending for the affected file. It will not actually flush those
> writes out, though. Instead, the pages are mapped read-only with a
> special flag, forcing a page fault when the process first attempts to
> perform a write to one of those pages. The fault handler will then
> flush out any dirty metadata synchronously, set the page permissions
> to allow the write, and return. At that point, the process can write
> the page safely, since all the necessary metadata changes have already
> made it to persistent storage.
Looking here:
https://software.intel.com/content/www/us/en/develop/articles/java-support-for-intel-optane-dc-persistent-memory.html#inpage-nav-4-2
It seems like Optane supports access via byte buffers - so I think
combining that with the new MapMode, and with the new memory segment API
should (hopefully) "just work".
Maurizio
On 25/06/2020 11:19, Johannes Lichtenberger wrote:
> Oh, and an additional question, as I'm not so much into I/O stuff as
> of now, sadly.
>
> I've read a lot about byte-addressable NVM / Intel Optane NVM. I think
> it's a perfect match for my project, as SirixDB is always appending
> revisions of data, never overwriting anything. First, during shadow
> copying of the affected leaf-to-root path of a database page, the
> database page itself gets versioned through a sliding snapshot
> algorithm. Thus, only modified records + records which fall out of the
> sliding window will be copied. Thus, it's great, that we can now read
> at least in 256 byte-chunks on Intel Optane NVM.
>
> I guess at least with the block-based API I could simply store files
> on an Intel Optane NVM device (not the SSDs). However, I wonder how it
> can be used as non-volatile RAM and how it stores stuff durably. I've
> read that you don't have to serialize to byte-arrays anymore, but the
> processor has to flush it's caches and you have to make sure that the
> data-structure is always consistent.
>
> I'd love to change SirixDB and tune it for this non-volatile memory,
> but I'm not sure how to proceed here.
>
> Kind regards
> Johannes
>
> Am Do., 25. Juni 2020 um 12:02 Uhr schrieb Johannes Lichtenberger
> <lichtenberger.johannes at gmail.com
> <mailto:lichtenberger.johannes at gmail.com>>:
>
> Thanks, is it already available in a JDK 14 binary or should I
> wait till october? Furthermore, regarding the usage itself, is
> this okay to get a byte-array from the memory mapped file or
> should I use some sort of MemoryLayout...?
>
> final byte[]page =new byte[dataLength];
>
> final VarHandle byteVarHandle =MemoryHandles.varHandle(byte.class,ByteOrder.nativeOrder());
>
> for (int i =0; i <dataLength; i++) {
> // 4 (dataLength) + 1 = 5 in offset, as the loop starts with 0 we
> have to add one to the offset page[i] = (byte)byteVarHandle.get(baseAddress.addOffset(5 + i), i);
> }
>
> And from what I understand about memory mapped files -- not
> related to your API -- but in general it makes sense to map the
> whole "database"-file and not just a small page-portion, right? As
> it will fetch the page on-demand into the non heap memory location.
>
> Another thing, which might be sonsense as of now is this I guess:
>
> https://github.com/sirixdb/sirix/blob/7bdbd5d17a034f02902f8f7dd0ef7012d89c81fb/bundles/sirix-core/src/main/java/org/sirix/io/memorymapped/MemoryMappedFileWriter.java#L140
>
>
> First, an (on-heap) byte-array is produced, such that a database
> page-fragment is serialized with all its records to it. Basically,
> the DataOutput interface is used to serialize/deserialize stuff
> in-memory, do compression, encryption... Changing this might be
> more work, and I guess it's okay.
>
> Then, I think I can skip the ByteBuffer (just copied from the
> RandomAccessFile based implementation), then copying the byte
> buffer again to the memory mapped file segment.
>
> Kind regards
>
> Johannes
>
>
> Am Do., 25. Juni 2020 um 11:26 Uhr schrieb Maurizio Cimadamore
> <maurizio.cimadamore at oracle.com
> <mailto:maurizio.cimadamore at oracle.com>>:
>
> Hi Johannes,
> the specific condition you are talking about has been
> rectified in the
> upcoming Java 15.
>
> The new code should be doing something like this:
>
> ```
> if (bytesSize < 0) throw new
> IllegalArgumentException("Requested bytes
> size must be >= 0.");
> if (bytesOffset < 0) throw new
> IllegalArgumentException("Requested bytes
> offset must be >= 0.");
> ```
>
> So, hopefully, your code should work against the latest
> version of the API
>
> Maurizio
>
> On 25/06/2020 08:24, Johannes Lichtenberger wrote:
> > Hi Paul,
> >
> > that's great. I guess usually there's a -dev and a -user
> Mailinglist,
> > that's why I thought it's for internal development topics.
> >
> > I'm using AdpotOpenJDK, because I wanted to try Shenandoah,
> which is at
> > least not in the Oracle binaries.
> >
> > IMPLEMENTOR="AdoptOpenJDK"
> > IMPLEMENTOR_VERSION="AdoptOpenJDK"
> > JAVA_VERSION="14.0.1"
> > JAVA_VERSION_DATE="2020-04-14"
> >
> > BTW: My OS is Linux/Ubuntu.
> >
> > Kind regards
> > Johannes
> >
> > Am Do., 25. Juni 2020 um 02:04 Uhr schrieb Paul Sandoz <
> > paul.sandoz at oracle.com <mailto:paul.sandoz at oracle.com>>:
> >
> >> Hi Johannes,
> >>
> >> This is the correct list. Feedback on the API and its
> usability is very
> >> important, as well as whether the implementation works as
> expected.
> >>
> >> This was fixed in
> https://bugs.openjdk.java.net/browse/JDK-8246095
> >>
> >>
> >>
> https://github.com/openjdk/panama-foreign/blob/foreign-memaccess/src/jdk.incubator.foreign/share/classes/jdk/internal/foreign/MappedMemorySegmentImpl.java#L102
> >>
> >> What JDK build are you using?
> >>
> >> Paul.
> >>
> >> On Jun 24, 2020, at 3:48 PM, Johannes Lichtenberger <
> >> lichtenberger.johannes at gmail.com
> <mailto:lichtenberger.johannes at gmail.com>> wrote:
> >>
> >> Hi,
> >>
> >> I guess that's not the right mailinglist to ask for help
> with the API, but
> >> maybe you can point me at least somewhere.
> >>
> >> Currently I face the problem, that I can't get a memory
> segment on an empty
> >> file when I want to memory map a file. However, a writer
> would write to the
> >> memory mapped file.
> >>
> >> I want to be able to swap a rather old RandomAccessFile based
> >> implementation with a memory mapped file implementation.
> The two classes
> >> (actually three, but only the FileReader and FileWriter
> matters) are rather
> >> simple.
> >>
> >> The old package:
> >>
> >>
> >>
> https://github.com/sirixdb/sirix/tree/master/bundles/sirix-core/src/main/java/org/sirix/io/file
> >>
> >>
> >> and the new package
> >>
> >>
> >>
> https://github.com/sirixdb/sirix/tree/master/bundles/sirix-core/src/main/java/org/sirix/io/memorymapped
> >>
> >> This line fails if the underlying file size is 0:
> >>
> >>
> >>
> https://github.com/sirixdb/sirix/blob/7bdbd5d17a034f02902f8f7dd0ef7012d89c81fb/bundles/sirix-core/src/main/java/org/sirix/io/memorymapped/MemoryMappedFileWriter.java#L83
> >>
> >> Stacktrace (Precondition fails, so I'm not using the API
> in the intended
> >> way):
> >>
> >> java.lang.IllegalArgumentException: Requested bytes size
> must be > 0.
> >>
> >> at
> >>
> >>
> jdk.incubator.foreign/jdk.internal.foreign.Utils.makeMappedSegment(Utils.java:140)
> >> at
> >>
> >>
> jdk.incubator.foreign/jdk.incubator.foreign.MemorySegment.mapFromPath(MemorySegment.java:398)
> >> at
> >>
> >>
> org.sirix.io.memorymapped.MemoryMappedFileWriter.<init>(MemoryMappedFileWriter.java:83)
> >>
> >>
> >> So, it might be easiest to write a more or less direct
> conversion to the
> >> Foreign Memory API. Maybe someone can help or point me to a
> helpful
> >> Mailinglist/Forum/whatever.
> >>
> >> Kind regards
> >> Johannes
> >>
> >>
> >>
>
More information about the panama-dev
mailing list