Memory Mapped file segment (file is empty)

Thu Jun 25 10:19:31 UTC 2020

Oh, and an additional question, as I'm not so much into I/O stuff as of
now, sadly.

I've read a lot about byte-addressable NVM / Intel Optane NVM. I think it's
a perfect match for my project, as SirixDB is always appending revisions of
data, never overwriting anything. First, during shadow copying of the
affected leaf-to-root path of a database page, the database page itself
gets versioned through a sliding snapshot algorithm. Thus, only modified
records + records which fall out of the sliding window will be copied.
Thus, it's great, that we can now read at least in 256 byte-chunks on Intel
Optane NVM.

I guess at least with the block-based API I could simply store files on an
Intel Optane NVM device (not the SSDs). However, I wonder how it can be
used as non-volatile RAM and how it stores stuff durably. I've read that
you don't have to serialize to byte-arrays anymore, but the processor has
to flush it's caches and you have to make sure that the data-structure is
always consistent.

I'd love to change SirixDB and tune it for this non-volatile memory, but
I'm not sure how to proceed here.

Kind regards
Johannes

Am Do., 25. Juni 2020 um 12:02 Uhr schrieb Johannes Lichtenberger <
lichtenberger.johannes at gmail.com>:

> Thanks, is it already available in a JDK 14 binary or should I wait till
> october? Furthermore, regarding the usage itself, is this okay to get a
> byte-array from the memory mapped file or should I use some sort of
> MemoryLayout...?
>
> final byte[] page = new byte[dataLength];
>
> final VarHandle byteVarHandle = MemoryHandles.varHandle(byte.class, ByteOrder.nativeOrder());
>
> for (int i = 0; i < dataLength; i++) {
>   // 4 (dataLength) + 1 = 5 in offset, as the loop starts with 0 we have to add one to the offset
>   page[i] = (byte) byteVarHandle.get(baseAddress.addOffset(5 + i), i);
> }
>
>
> And from what I understand about memory mapped files -- not related to your API -- but in general it makes sense to map the whole "database"-file and not just a small page-portion, right? As it will fetch the page on-demand into the non heap memory location.
>
> Another thing, which might be sonsense as of now is this I guess:
>
>
> https://github.com/sirixdb/sirix/blob/7bdbd5d17a034f02902f8f7dd0ef7012d89c81fb/bundles/sirix-core/src/main/java/org/sirix/io/memorymapped/MemoryMappedFileWriter.java#L140
>
>
> First, an (on-heap) byte-array is produced, such that a database page-fragment is serialized with all its records to it. Basically, the DataOutput interface is used to serialize/deserialize stuff in-memory, do compression, encryption... Changing this might be more work, and I guess it's okay.
>
> Then, I think I can skip the ByteBuffer (just copied from the RandomAccessFile based implementation), then copying the byte buffer again to the memory mapped file segment.
>
>
> Kind regards
>
> Johannes
>
>
> Am Do., 25. Juni 2020 um 11:26 Uhr schrieb Maurizio Cimadamore <
> maurizio.cimadamore at oracle.com>:
>
>> Hi Johannes,
>> the specific condition you are talking about has been rectified in the
>> upcoming Java 15.
>>
>> The new code should be doing something like this:
>>
>> ```
>> if (bytesSize < 0) throw new IllegalArgumentException("Requested bytes
>> size must be >= 0.");
>> if (bytesOffset < 0) throw new IllegalArgumentException("Requested bytes
>> offset must be >= 0.");
>> ```
>>
>> So, hopefully, your code should work against the latest version of the API
>>
>> Maurizio
>>
>> On 25/06/2020 08:24, Johannes Lichtenberger wrote:
>> > Hi Paul,
>> >
>> > that's great. I guess usually there's a -dev and a -user Mailinglist,
>> > that's why I thought it's for internal development topics.
>> >
>> > I'm using AdpotOpenJDK, because I wanted to try Shenandoah, which is at
>> > least not in the Oracle binaries.
>> >
>> > IMPLEMENTOR="AdoptOpenJDK"
>> > IMPLEMENTOR_VERSION="AdoptOpenJDK"
>> > JAVA_VERSION="14.0.1"
>> > JAVA_VERSION_DATE="2020-04-14"
>> >
>> > BTW: My OS is Linux/Ubuntu.
>> >
>> > Kind regards
>> > Johannes
>> >
>> > Am Do., 25. Juni 2020 um 02:04 Uhr schrieb Paul Sandoz <
>> > paul.sandoz at oracle.com>:
>> >
>> >> Hi Johannes,
>> >>
>> >> This is the correct list. Feedback on the API and its usability is very
>> >> important, as well as whether the implementation works as expected.
>> >>
>> >> This was fixed in https://bugs.openjdk.java.net/browse/JDK-8246095
>> >>
>> >>
>> >>
>> https://github.com/openjdk/panama-foreign/blob/foreign-memaccess/src/jdk.incubator.foreign/share/classes/jdk/internal/foreign/MappedMemorySegmentImpl.java#L102
>> >>
>> >> What JDK build are you using?
>> >>
>> >> Paul.
>> >>
>> >> On Jun 24, 2020, at 3:48 PM, Johannes Lichtenberger <
>> >> lichtenberger.johannes at gmail.com> wrote:
>> >>
>> >> Hi,
>> >>
>> >> I guess that's not the right mailinglist to ask for help with the API,
>> but
>> >> maybe you can point me at least somewhere.
>> >>
>> >> Currently I face the problem, that I can't get a memory segment on an
>> empty
>> >> file when I want to memory map a file. However, a writer would write
>> to the
>> >> memory mapped file.
>> >>
>> >> I want to be able to swap a rather old RandomAccessFile based
>> >> implementation with a memory mapped file implementation. The two
>> classes
>> >> (actually three, but only the FileReader and FileWriter matters) are
>> rather
>> >> simple.
>> >>
>> >> The old package:
>> >>
>> >>
>> >>
>> https://github.com/sirixdb/sirix/tree/master/bundles/sirix-core/src/main/java/org/sirix/io/file
>> >>
>> >>
>> >> and the new package
>> >>
>> >>
>> >>
>> https://github.com/sirixdb/sirix/tree/master/bundles/sirix-core/src/main/java/org/sirix/io/memorymapped
>> >>
>> >> This line fails if the underlying file size is 0:
>> >>
>> >>
>> >>
>> https://github.com/sirixdb/sirix/blob/7bdbd5d17a034f02902f8f7dd0ef7012d89c81fb/bundles/sirix-core/src/main/java/org/sirix/io/memorymapped/MemoryMappedFileWriter.java#L83
>> >>
>> >> Stacktrace  (Precondition fails, so I'm not using the API in the
>> intended
>> >> way):
>> >>
>> >> java.lang.IllegalArgumentException: Requested bytes size must be > 0.
>> >>
>> >> at
>> >>
>> >>
>> jdk.incubator.foreign/jdk.internal.foreign.Utils.makeMappedSegment(Utils.java:140)
>> >> at
>> >>
>> >>
>> jdk.incubator.foreign/jdk.incubator.foreign.MemorySegment.mapFromPath(MemorySegment.java:398)
>> >> at
>> >>
>> >>
>> org.sirix.io.memorymapped.MemoryMappedFileWriter.<init>(MemoryMappedFileWriter.java:83)
>> >>
>> >>
>> >> So, it might be easiest to write a more or less direct conversion to
>> the
>> >> Foreign Memory API. Maybe someone can help or point me to a helpful
>> >> Mailinglist/Forum/whatever.
>> >>
>> >> Kind regards
>> >> Johannes
>> >>
>> >>
>> >>
>>
>