can I expect a lock functionality on a MemorySegment similar to mlock?

[Thomas Stüfe]

Hi,

just a hopefully not stupid question:

Is it possible to use huge pages (oc Linux only) underlying a
MemorySegment?
If so, and if performance considerations are the motivator behind the wish
for mlock(), that could be a better choice.

Cheers, Thomas

On Mon, Oct 12, 2020 at 3:15 PM Maurizio Cimadamore <
maurizio.cimadamore at oracle.com> wrote:

>
> On 12/10/2020 12:17, kant kodali wrote:
> > yes but can I expect an API in the future that covers a good amount of
> > syscalls (similar to the Golang one).
>
> I believe the tools we're putting forward with the Foreign Memory Access
> and the Foreign Linker API, combined, make it trivial to provide an API
> such as the one you describe. Whether such an API will (or should) be
> provided from Oracle, in the form of a Java SE API, that's a separate
> question - and that's not where the priority lies right now. Right now
> the important thing, for us, is to make things like these *possible*.
>
> Maurizio
>
> >
> > On Mon, Oct 12, 2020 at 3:54 AM Maurizio Cimadamore
> > <maurizio.cimadamore at oracle.com
> > <mailto:maurizio.cimadamore at oracle.com>> wrote:
> >
> >
> >     On 12/10/2020 11:45, kant kodali wrote:
> >>     Thanks for the response. I will check out the Linker API and that
> >>     does look better than JNI however, it still looks very clunky but
> >>     I understand introducing something like mlock would break the
> >>     "write once run anywhere" principle. I personally don't see harm
> >>     in supporting something that is OS specific as long as the user
> >>     is well aware. For example, say if you use package X there is no
> >>     guarantee of "write once run anywhere" I would be totally
> >>     fine with that. Currently as you were saying there is some way to
> >>     call mlock like using the Linker API so why not a cleaner API
> >>     like this https://golang.org/pkg/syscall/
> >>     <
> https://urldefense.com/v3/__https://golang.org/pkg/syscall/__;!!GqivPVa7Brio!LdF8xYjuFb09L7U2EZPea81Mk-FmSIalOoNISokPl0BGjdUPRoRvjQzG1GxFI6gUbUMTI-w$
> >
> >>     ? It is purely in Golang and looks so simple to use for any low
> >>     level application.
> >
> >     Without diving too much into the specifics, reading from the docs,
> >     it doesn't seem to me as if `syscall` supports "write once and run
> >     anywhere" - this document:
> >
> >
> https://docs.google.com/document/d/1QXzI9I1pOfZPujQzxhyRy6EeHYTQitKKjHfpq0zpxZs/edit
> >     <
> https://urldefense.com/v3/__https://docs.google.com/document/d/1QXzI9I1pOfZPujQzxhyRy6EeHYTQitKKjHfpq0zpxZs/edit__;!!GqivPVa7Brio!Lm0SKA0AOMaQvNO2Ef0aKpdyPoYAHvQP6CnAN-487WcldE8WyFSJXoaxrz-LurUrRw370U0$
> >
> >
> >     Seems to point out at the need to separate out the functionalities
> >     in different packages, windows, linux, etc - which is more or less
> >     what I'd expect for such a low level set of functionalities.
> >
> >     Note that the foreign linker method handles are a "tool" - not a
> >     final API. You can write a class which exports some kind of
> >     `mlock` Java method which, internally might use method handle X on
> >     Linux and method handle Y on Windows (or some other code). So,
> >     while the _implementation_ of the `mlock` method will have to
> >     (necessarily) distinguish between platforms, clients of this
> >     method won't have to.
> >
> >     Maurizio
> >
> >
> >>
> >>
> >>
> >>     On Mon, Oct 12, 2020 at 2:47 AM Maurizio Cimadamore
> >>     <maurizio.cimadamore at oracle.com
> >>     <mailto:maurizio.cimadamore at oracle.com>> wrote:
> >>
> >>         We have no plans to add such a functionality directly, in the
> >>         memory
> >>         segment API. However, with the addition of the Foreign Linker
> >>         API,
> >>         things are not so bad, you can define your mlock MethodHandle
> >>         once and
> >>         for all:
> >>
> >>         ```
> >>         //int mlock(const void *addr, size_t len);
> >>         MethodHandle mlock = CLinker.getInstance().downcallHandle(
> >>         LibraryLookup.ofDefault().lookup("mlock").get(),
> >>                  MethodType.methodType(int.class,
> >>         MemoryAddress.class, long.class),
> >>                  FunctionDescriptor.of(C_INT, C_POINTER, C_LONG)
> >>         );
> >>         ```
> >>
> >>         And then you can call it on an existing segment, as desired:
> >>
> >>         ```
> >>         MemorySegment segment = ...
> >>         mlock.invokeExact(segment.address(), 100L);
> >>         ```
> >>
> >>         No JNI code is required to do this; so clients might define
> >>         all the
> >>         helper method handles they need to interact with mapped
> >>         memory e.g. in a
> >>         separate "util" class.
> >>
> >>         Full disclosure: I'm not a super fan of having such low level
> >>         functionality API points; learning from our experience with
> >>         existing
> >>         similar low-level functionalities, such as mapped buffer/segment
> >>         force/load/unload, what I've learned is that an approach that
> >>         tries to
> >>         expose a single API, regardless of the platform is not a very
> >>         sharp
> >>         tool, and you end up with things like (see Windows
> >>         implementation of
> >>         MappedMemoryUtils.c):
> >>
> >>         ```
> >>         NIEXPORT void JNICALL
> >>         Java_java_nio_MappedMemoryUtils_load0(JNIEnv *env, jobject
> >>         obj, jlong
> >>         address,
> >>                                               jlong len)
> >>         {
> >>              // no madvise available
> >>         }
> >>         ```
> >>
> >>         I believe a demand for such capabilities, in the past, has
> >>         come from the
> >>         fact that JNI is the only tool which allows you to peek and
> >>         poke at
> >>         mapped memory; that put increased demand for such functions
> >>         in the
> >>         public ByteBuffer API. But as you can clearly see, you can't
> >>         expect
> >>         these function to behave correctly on all systems - and even
> >>         within a
> >>         "supported" system, different clients and workload, might
> >>         need different
> >>         tuning as to how the memory mapping is implemented.
> >>
> >>         As a result, my feeling is that we could spend years trying
> >>         to add a
> >>         rich set of memory mapping functions, without really
> >>         improving much over
> >>         the status quo. If you are a client where the use of "mlock"
> >>         is crucial,
> >>         I think you should just tap into that - and I think the
> >>         linker API makes
> >>         it easy enough for you to do that w/o resorting to any magic JNI
> >>         incantation.
> >>
> >>         Cheers
> >>         Maurizio
> >>
> >>
> >>         On 11/10/2020 11:43, kant kodali wrote:
> >>         > Hi All,
> >>         >
> >>         > I am implementing a buffer pool for my experimental
> >>         database and ideally I
> >>         > want to be able to pin a page using Java(without using
> >>         JNI/JNA myself) and
> >>         > by pinning a page I mean prevent a page from being swapped
> >>         out or in. some
> >>         > java version of mlock system call would work so I was
> >>         thinking why not a
> >>         > mlock like functionality on MemorySegment? If I have to
> >>         resort to JNI/JNA
> >>         > all the time then it makes me think Java shouldn't be the
> >>         right language to
> >>         > build a database like application. please let me know what
> >>         you think?
> >>         >
> >>         > Thanks,
> >>         > Kant
> >>
>