Pinning of on-heap MemorySegment

Mon Aug 28 14:21:36 UTC 2023

I think that's what I was referring to w.r.t. "scope of pinning".

We have investigated doing things like:

```
MemorySegment heapBytes = MemorySegment.ofArray(new byte[...]);
try (Arena arena = ...) {
     MemorySegment pinnedBytes = heapBytes.pin(arena);
     ...
} // unpinned here
```

This works well API-wise (and it gives you all the lifetime management 
options you'd come to expect when interaging with a segment), but this 
could easily lead to Java programs blocking GC indefinitively, which is 
a risk that has to be factored in. Yes, we could make MS::pin a 
restricted method, (I see that C# went down the unsafe path too), but 
the question is whether that's enough of a deterrent, given how much 
accidental damage you one cause (w/o realizing).

Maurizio

On 28/08/2023 15:09, Gavin Ray wrote:
> For whatever it's worth, I think that C# does pinning fairly well.
> It uses a "fixed" statement around a variable which is similar to a 
> try-with-resources in Java.
>
> Without introducing a new keyword this could probably be done as a 
> Functional Interface that takes a lamba or something.
>
> > withPinned(heapSegment, segment -> {});
>
> - fixed statement - pin a moveable variable | Microsoft Learn 
> <https://urldefense.com/v3/__https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/statements/fixed__;!!ACWV5N9M2RV99hQ!IGKbapxVG_QMwdL236-U-1XDKQLgz1imop4dEpsa2CqyUkuhuLs7lk5LYrCbzu3nH6uT0YTrJVKY329M7mInwa_nqzLD$>
> - Copying and Pinning - .NET Framework | Microsoft Learn 
> <https://urldefense.com/v3/__https://learn.microsoft.com/en-us/dotnet/framework/interop/copying-and-pinning__;!!ACWV5N9M2RV99hQ!IGKbapxVG_QMwdL236-U-1XDKQLgz1imop4dEpsa2CqyUkuhuLs7lk5LYrCbzu3nH6uT0YTrJVKY329M7mInweV3GGr6$>
>
> On Mon, Aug 28, 2023 at 9:04 AM Yasumasa Suenaga 
> <suenaga at oss.nttdata.com> wrote:
>
>     Hi Maurizio,
>
>     Thank you for knowing ffmasm :)
>
>     MS::pin/unpin in my proposal are same semantics with
>     GetPrimitiveArrayCritical/ReleasePrimitiveArrayCritical in JNI. They
>     might not be leveraged a lot of developers, but they help a part of
>     experienced developers to improve their application performance.
>     In JNI,
>     we call GetPrimitiveArrayCritical() to access large arrays without
>     memory copy cost. It makes sence. I just want to do same operation in
>     FFM.
>
>     Let's think about image processing. When we want to binarize JPEG
>     image,
>     we would load original image with ImageIO API. Then we can get pixels
>     from BufferedImage::getRGB() as int[]. So we have to copy pixels into
>     off-heap MemorySegment if we want to perform operations in native
>     (e.g.
>     SIMD, GPU processing). I'm sure most of Java API / third party
>     libraries
>     use primitive array, not MemorySegment. So I think it is better if we
>     pin on-heap memory and leverage it in native.
>
>     Of course I understand it might be some worse behaviors especially
>     GC,
>     so pinning might not be recommended for all of Java developers. But I
>     believe pinning is welcomed from developers who experienced in native
>     (C/C++/assembly/GPGPU and so on) because they want to offload
>     processing
>     which needs large memory (preprocessed in Java).
>
>
>     Thanks,
>
>     Yasumasa
>
>
>     2023-08-28 18:48 に Maurizio Cimadamore さんは書きました:
>     > On 28/08/2023 08:29, Yasumasa Suenaga wrote:
>     >> Hi all,
>     >>
>     >> I'm very interested in FFM, especially generate assembly code
>     in Java
>     >> and calling them with FFM [1].
>     > Hi,
>     > I've seen your work a year or so ago and have been very
>     impressed by it
>     > :-)
>     >>
>     >> I think one of the performance neck is MemorySegment because
>     all of
>     >> on-heap regions should be copied into off-heap region (native
>     segment)
>     >> when they are referred from foreign function. So I'm expecting to
>     >> implement sort of pinning operation for on-heap MemorySegment
>     like a
>     >> JNI. I guess it is mentioned in FFM update in last month [2],
>     however
>     >> "Pinning of heap segments" does not have any links - I guess
>     nobody is
>     >> working for this yet. Do you have any updates for pinning?
>     >>
>     >> I've played them with both OpenJDK 22 with pinning support [3] and
>     >> ffmasm (hand-assembler for Java powered by FFM) [4]. I added
>     pin/unpin
>     >> method into Unsafe, and they are called by HeapMemorySegmentImpl.
>     >> Finally I got about 16x performance gain compared to
>     non-pinning code
>     >> [5] on my laptop.
>     >>
>     >> I guess FFM API reaches to goal step by step, but it is still a
>     >> preview in JDK 21. I hope that pinning feature is supported
>     into JDK
>     >> 22 because I believe we can leverage FFM for more fields! I'm
>     happy to
>     >> contribute/help to implement pinning feature if it needs.
>     >
>     > While there's no doubt that in some applications and use cases
>     pinning
>     > provides a significant performance boost, there are some challenges:
>     >
>     > 1. scope of pinning: is pinning allowed on a per-native-call
>     basis? Or
>     > is it something more general?
>     > 2. does the garbage collection support region-based pinning [1] ?
>     > 3. why is pinning needed in the first place?
>     >
>     > (1) and (2) are very much linked. Not all garbage collectors support
>     > fine-grained pinning mechanism. Which means that, in most of
>     them, if
>     > you pin, you effectively block GC for the entire duration of the pin
>     > operation (GC locker mechanism). This is something that, as I'm sure
>     > you understand, is not very desirable. For this reason, it might be
>     > better to consider a pinning API which only pins for the
>     duration of a
>     > native call (e.g. in the shape of an additional linker option).
>     While
>     > FFM could support more complex pinning policies (e.g. pin a segment
>     > inside an Arena, so that segment is unpinned when the arena is
>     > closed), given the uneven support for fine-grained pinning
>     across GCs,
>     > I'm not sure such a general API (which is similar to your "MS::pin"
>     > method) would be a good idea. We're doing some experiments for
>     adding
>     > a new linker option which allows for pin heap segments to be pinned
>     > when calling a downcall method handle, we're not yet sure of its
>     > inclusion, but it would be something worth publishing somewhere
>     (when
>     > ready) so that developers (like you) can play with it and provide
>     > feedback.
>     >
>     > Then there's (3). Most of the times, pinning is used in order to
>     > interact with native calls from public-facing APIs that are "stuck"
>     > using array syntax. That is, in order to be user friendly, such API
>     > work with arrays - but then a problem arises when trying to use the
>     > contents of the array off-heap. But what if the memory was
>     off-heap to
>     > begin with? Then no memory copy would be required. I believe the
>     > biggest impediment for off-heap memory being used directly has to do
>     > with the fact that, for users, interacting with an `int[]` is
>     > significantly easier than interacting with a `MemorySegment`, or a
>     > `ByteBuffer`. But what if we could provide some mechanism to
>     create an
>     > "array view" over an off-heap memory region? Now clients would
>     be able
>     > to use the beloved `[]` syntax, even if memory access remained
>     > off-heap.
>     >
>     > While we don't have any concrete proposal on this latter point,
>     we do
>     > believe that the topic of making memory segments (or byte buffer)
>     > easier to access for "legacy clients" is inextricably linked to the
>     > topic of pinning of heap memory.
>     >
>     > [1] - https://openjdk.org/jeps/423
>     >
>     >>
>     >>
>     >> Thanks,
>     >>
>     >> Yasumasa
>     >>
>     >>
>     >> [1] https://github.com/YaSuenag/ffmasm
>     <https://urldefense.com/v3/__https://github.com/YaSuenag/ffmasm__;!!ACWV5N9M2RV99hQ!IGKbapxVG_QMwdL236-U-1XDKQLgz1imop4dEpsa2CqyUkuhuLs7lk5LYrCbzu3nH6uT0YTrJVKY329M7mInwXvCTRxW$>
>     >> [2]
>     >> https://mail.openjdk.org/pipermail/panama-dev/2023-July/019510.html
>     >> [3]
>     >>
>     https://github.com/YaSuenag/jdk/commit/f0a9b3705b3ecdf3dbb6b80cac9d53456f08f967
>     <https://urldefense.com/v3/__https://github.com/YaSuenag/jdk/commit/f0a9b3705b3ecdf3dbb6b80cac9d53456f08f967__;!!ACWV5N9M2RV99hQ!IGKbapxVG_QMwdL236-U-1XDKQLgz1imop4dEpsa2CqyUkuhuLs7lk5LYrCbzu3nH6uT0YTrJVKY329M7mInwbe47VCY$>
>     >> [4]
>     >>
>     https://github.com/YaSuenag/ffmasm/commit/925608538b936db1b311ae84e12fa0252058b7f4
>     <https://urldefense.com/v3/__https://github.com/YaSuenag/ffmasm/commit/925608538b936db1b311ae84e12fa0252058b7f4__;!!ACWV5N9M2RV99hQ!IGKbapxVG_QMwdL236-U-1XDKQLgz1imop4dEpsa2CqyUkuhuLs7lk5LYrCbzu3nH6uT0YTrJVKY329M7mInwbUJ6qi5$>
>     >> [5]
>     >>
>     https://github.com/YaSuenag/ffmasm/blob/ffm-pinning/benchmarks/vectorapi/src/main/java/com/yasuenag/ffmasm/benchmark/vectorapi/VectorOpComparison.java
>     <https://urldefense.com/v3/__https://github.com/YaSuenag/ffmasm/blob/ffm-pinning/benchmarks/vectorapi/src/main/java/com/yasuenag/ffmasm/benchmark/vectorapi/VectorOpComparison.java__;!!ACWV5N9M2RV99hQ!IGKbapxVG_QMwdL236-U-1XDKQLgz1imop4dEpsa2CqyUkuhuLs7lk5LYrCbzu3nH6uT0YTrJVKY329M7mInwRLDJMg-$>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/panama-dev/attachments/20230828/d6231ae4/attachment-0001.htm>