Pinning of on-heap MemorySegment

[Maurizio Cimadamore]

On 28/08/2023 08:29, Yasumasa Suenaga wrote:
> Hi all,
>
> I'm very interested in FFM, especially generate assembly code in Java 
> and calling them with FFM [1].
Hi,
I've seen your work a year or so ago and have been very impressed by it :-)
>
> I think one of the performance neck is MemorySegment because all of 
> on-heap regions should be copied into off-heap region (native segment) 
> when they are referred from foreign function. So I'm expecting to 
> implement sort of pinning operation for on-heap MemorySegment like a 
> JNI. I guess it is mentioned in FFM update in last month [2], however 
> "Pinning of heap segments" does not have any links - I guess nobody is 
> working for this yet. Do you have any updates for pinning?
>
> I've played them with both OpenJDK 22 with pinning support [3] and 
> ffmasm (hand-assembler for Java powered by FFM) [4]. I added pin/unpin 
> method into Unsafe, and they are called by HeapMemorySegmentImpl. 
> Finally I got about 16x performance gain compared to non-pinning code 
> [5] on my laptop.
>
> I guess FFM API reaches to goal step by step, but it is still a 
> preview in JDK 21. I hope that pinning feature is supported into JDK 
> 22 because I believe we can leverage FFM for more fields! I'm happy to 
> contribute/help to implement pinning feature if it needs.

While there's no doubt that in some applications and use cases pinning 
provides a significant performance boost, there are some challenges:

1. scope of pinning: is pinning allowed on a per-native-call basis? Or 
is it something more general?
2. does the garbage collection support region-based pinning [1] ?
3. why is pinning needed in the first place?

(1) and (2) are very much linked. Not all garbage collectors support 
fine-grained pinning mechanism. Which means that, in most of them, if 
you pin, you effectively block GC for the entire duration of the pin 
operation (GC locker mechanism). This is something that, as I'm sure you 
understand, is not very desirable. For this reason, it might be better 
to consider a pinning API which only pins for the duration of a native 
call (e.g. in the shape of an additional linker option). While FFM could 
support more complex pinning policies (e.g. pin a segment inside an 
Arena, so that segment is unpinned when the arena is closed), given the 
uneven support for fine-grained pinning across GCs, I'm not sure such a 
general API (which is similar to your "MS::pin" method) would be a good 
idea. We're doing some experiments for adding a new linker option which 
allows for pin heap segments to be pinned when calling a downcall method 
handle, we're not yet sure of its inclusion, but it would be something 
worth publishing somewhere (when ready) so that developers (like you) 
can play with it and provide feedback.

Then there's (3). Most of the times, pinning is used in order to 
interact with native calls from public-facing APIs that are "stuck" 
using array syntax. That is, in order to be user friendly, such API work 
with arrays - but then a problem arises when trying to use the contents 
of the array off-heap. But what if the memory was off-heap to begin 
with? Then no memory copy would be required. I believe the biggest 
impediment for off-heap memory being used directly has to do with the 
fact that, for users, interacting with an `int[]` is significantly 
easier than interacting with a `MemorySegment`, or a `ByteBuffer`. But 
what if we could provide some mechanism to create an "array view" over 
an off-heap memory region? Now clients would be able to use the beloved 
`[]` syntax, even if memory access remained off-heap.

While we don't have any concrete proposal on this latter point, we do 
believe that the topic of making memory segments (or byte buffer) easier 
to access for "legacy clients" is inextricably linked to the topic of 
pinning of heap memory.

[1] - https://openjdk.org/jeps/423

>
>
> Thanks,
>
> Yasumasa
>
>
> [1] https://github.com/YaSuenag/ffmasm
> [2] https://mail.openjdk.org/pipermail/panama-dev/2023-July/019510.html
> [3] 
> https://github.com/YaSuenag/jdk/commit/f0a9b3705b3ecdf3dbb6b80cac9d53456f08f967
> [4] 
> https://github.com/YaSuenag/ffmasm/commit/925608538b936db1b311ae84e12fa0252058b7f4
> [5] 
> https://github.com/YaSuenag/ffmasm/blob/ffm-pinning/benchmarks/vectorapi/src/main/java/com/yasuenag/ffmasm/benchmark/vectorapi/VectorOpComparison.java