can I expect a lock functionality on a MemorySegment similar to mlock?

[Maurizio Cimadamore]

On 12/10/2020 12:17, kant kodali wrote:
> yes but can I expect an API in the future that covers a good amount of 
> syscalls (similar to the Golang one).

I believe the tools we're putting forward with the Foreign Memory Access 
and the Foreign Linker API, combined, make it trivial to provide an API 
such as the one you describe. Whether such an API will (or should) be 
provided from Oracle, in the form of a Java SE API, that's a separate 
question - and that's not where the priority lies right now. Right now 
the important thing, for us, is to make things like these *possible*.

Maurizio

>
> On Mon, Oct 12, 2020 at 3:54 AM Maurizio Cimadamore 
> <maurizio.cimadamore at oracle.com 
> <mailto:maurizio.cimadamore at oracle.com>> wrote:
>
>
>     On 12/10/2020 11:45, kant kodali wrote:
>>     Thanks for the response. I will check out the Linker API and that
>>     does look better than JNI however, it still looks very clunky but
>>     I understand introducing something like mlock would break the
>>     "write once run anywhere" principle. I personally don't see harm
>>     in supporting something that is OS specific as long as the user
>>     is well aware. For example, say if you use package X there is no
>>     guarantee of "write once run anywhere" I would be totally
>>     fine with that. Currently as you were saying there is some way to
>>     call mlock like using the Linker API so why not a cleaner API
>>     like this https://golang.org/pkg/syscall/
>>     <https://urldefense.com/v3/__https://golang.org/pkg/syscall/__;!!GqivPVa7Brio!LdF8xYjuFb09L7U2EZPea81Mk-FmSIalOoNISokPl0BGjdUPRoRvjQzG1GxFI6gUbUMTI-w$>
>>     ? It is purely in Golang and looks so simple to use for any low
>>     level application.
>
>     Without diving too much into the specifics, reading from the docs,
>     it doesn't seem to me as if `syscall` supports "write once and run
>     anywhere" - this document:
>
>     https://docs.google.com/document/d/1QXzI9I1pOfZPujQzxhyRy6EeHYTQitKKjHfpq0zpxZs/edit
>     <https://urldefense.com/v3/__https://docs.google.com/document/d/1QXzI9I1pOfZPujQzxhyRy6EeHYTQitKKjHfpq0zpxZs/edit__;!!GqivPVa7Brio!Lm0SKA0AOMaQvNO2Ef0aKpdyPoYAHvQP6CnAN-487WcldE8WyFSJXoaxrz-LurUrRw370U0$>
>
>     Seems to point out at the need to separate out the functionalities
>     in different packages, windows, linux, etc - which is more or less
>     what I'd expect for such a low level set of functionalities.
>
>     Note that the foreign linker method handles are a "tool" - not a
>     final API. You can write a class which exports some kind of
>     `mlock` Java method which, internally might use method handle X on
>     Linux and method handle Y on Windows (or some other code). So,
>     while the _implementation_ of the `mlock` method will have to
>     (necessarily) distinguish between platforms, clients of this
>     method won't have to.
>
>     Maurizio
>
>
>>
>>
>>
>>     On Mon, Oct 12, 2020 at 2:47 AM Maurizio Cimadamore
>>     <maurizio.cimadamore at oracle.com
>>     <mailto:maurizio.cimadamore at oracle.com>> wrote:
>>
>>         We have no plans to add such a functionality directly, in the
>>         memory
>>         segment API. However, with the addition of the Foreign Linker
>>         API,
>>         things are not so bad, you can define your mlock MethodHandle
>>         once and
>>         for all:
>>
>>         ```
>>         //int mlock(const void *addr, size_t len);
>>         MethodHandle mlock = CLinker.getInstance().downcallHandle(
>>         LibraryLookup.ofDefault().lookup("mlock").get(),
>>                  MethodType.methodType(int.class,
>>         MemoryAddress.class, long.class),
>>                  FunctionDescriptor.of(C_INT, C_POINTER, C_LONG)
>>         );
>>         ```
>>
>>         And then you can call it on an existing segment, as desired:
>>
>>         ```
>>         MemorySegment segment = ...
>>         mlock.invokeExact(segment.address(), 100L);
>>         ```
>>
>>         No JNI code is required to do this; so clients might define
>>         all the
>>         helper method handles they need to interact with mapped
>>         memory e.g. in a
>>         separate "util" class.
>>
>>         Full disclosure: I'm not a super fan of having such low level
>>         functionality API points; learning from our experience with
>>         existing
>>         similar low-level functionalities, such as mapped buffer/segment
>>         force/load/unload, what I've learned is that an approach that
>>         tries to
>>         expose a single API, regardless of the platform is not a very
>>         sharp
>>         tool, and you end up with things like (see Windows
>>         implementation of
>>         MappedMemoryUtils.c):
>>
>>         ```
>>         NIEXPORT void JNICALL
>>         Java_java_nio_MappedMemoryUtils_load0(JNIEnv *env, jobject
>>         obj, jlong
>>         address,
>>                                               jlong len)
>>         {
>>              // no madvise available
>>         }
>>         ```
>>
>>         I believe a demand for such capabilities, in the past, has
>>         come from the
>>         fact that JNI is the only tool which allows you to peek and
>>         poke at
>>         mapped memory; that put increased demand for such functions
>>         in the
>>         public ByteBuffer API. But as you can clearly see, you can't
>>         expect
>>         these function to behave correctly on all systems - and even
>>         within a
>>         "supported" system, different clients and workload, might
>>         need different
>>         tuning as to how the memory mapping is implemented.
>>
>>         As a result, my feeling is that we could spend years trying
>>         to add a
>>         rich set of memory mapping functions, without really
>>         improving much over
>>         the status quo. If you are a client where the use of "mlock"
>>         is crucial,
>>         I think you should just tap into that - and I think the
>>         linker API makes
>>         it easy enough for you to do that w/o resorting to any magic JNI
>>         incantation.
>>
>>         Cheers
>>         Maurizio
>>
>>
>>         On 11/10/2020 11:43, kant kodali wrote:
>>         > Hi All,
>>         >
>>         > I am implementing a buffer pool for my experimental
>>         database and ideally I
>>         > want to be able to pin a page using Java(without using
>>         JNI/JNA myself) and
>>         > by pinning a page I mean prevent a page from being swapped
>>         out or in. some
>>         > java version of mlock system call would work so I was
>>         thinking why not a
>>         > mlock like functionality on MemorySegment? If I have to
>>         resort to JNI/JNA
>>         > all the time then it makes me think Java shouldn't be the
>>         right language to
>>         > build a database like application. please let me know what
>>         you think?
>>         >
>>         > Thanks,
>>         > Kant
>>