Pinning of on-heap MemorySegment

Mon Aug 28 14:37:02 UTC 2023

Ah, that seems reasonable to me, being arena-scoped.

About the GC blocking -- I personally look at mucking about with Unsafe
things as an "all-bets-are-off" area.
If you step out of the bounds of the JVM GC and memory management, you're
on your own.

Where this feels like it could potentially be an issue is when consumers
interact with a library/API that utilizes something like pinning under the
hood.
Even if the user is aware of this, it seems like one thing that may make
sense is to make the ability to indefinitely pin segments gated by a flag.

That way the user has to knowingly opt-in to the potential of things going
awry?

On Mon, Aug 28, 2023 at 10:21 AM Maurizio Cimadamore <
maurizio.cimadamore at oracle.com> wrote:

> I think that's what I was referring to w.r.t. "scope of pinning".
>
> We have investigated doing things like:
>
> ```
> MemorySegment heapBytes = MemorySegment.ofArray(new byte[...]);
> try (Arena arena = ...) {
>     MemorySegment pinnedBytes = heapBytes.pin(arena);
>     ...
> } // unpinned here
> ```
>
> This works well API-wise (and it gives you all the lifetime management
> options you'd come to expect when interaging with a segment), but this
> could easily lead to Java programs blocking GC indefinitively, which is a
> risk that has to be factored in. Yes, we could make MS::pin a restricted
> method, (I see that C# went down the unsafe path too), but the question is
> whether that's enough of a deterrent, given how much accidental damage you
> one cause (w/o realizing).
>
> Maurizio
> On 28/08/2023 15:09, Gavin Ray wrote:
>
> For whatever it's worth, I think that C# does pinning fairly well.
> It uses a "fixed" statement around a variable which is similar to a
> try-with-resources in Java.
>
> Without introducing a new keyword this could probably be done as a
> Functional Interface that takes a lamba or something.
>
> > withPinned(heapSegment, segment -> {});
>
> - fixed statement - pin a moveable variable | Microsoft Learn
> <https://urldefense.com/v3/__https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/statements/fixed__;!!ACWV5N9M2RV99hQ!IGKbapxVG_QMwdL236-U-1XDKQLgz1imop4dEpsa2CqyUkuhuLs7lk5LYrCbzu3nH6uT0YTrJVKY329M7mInwa_nqzLD$>
> - Copying and Pinning - .NET Framework | Microsoft Learn
> <https://urldefense.com/v3/__https://learn.microsoft.com/en-us/dotnet/framework/interop/copying-and-pinning__;!!ACWV5N9M2RV99hQ!IGKbapxVG_QMwdL236-U-1XDKQLgz1imop4dEpsa2CqyUkuhuLs7lk5LYrCbzu3nH6uT0YTrJVKY329M7mInweV3GGr6$>
>
> On Mon, Aug 28, 2023 at 9:04 AM Yasumasa Suenaga <suenaga at oss.nttdata.com>
> wrote:
>
>> Hi Maurizio,
>>
>> Thank you for knowing ffmasm :)
>>
>> MS::pin/unpin in my proposal are same semantics with
>> GetPrimitiveArrayCritical/ReleasePrimitiveArrayCritical in JNI. They
>> might not be leveraged a lot of developers, but they help a part of
>> experienced developers to improve their application performance. In JNI,
>> we call GetPrimitiveArrayCritical() to access large arrays without
>> memory copy cost. It makes sence. I just want to do same operation in
>> FFM.
>>
>> Let's think about image processing. When we want to binarize JPEG image,
>> we would load original image with ImageIO API. Then we can get pixels
>> from BufferedImage::getRGB() as int[]. So we have to copy pixels into
>> off-heap MemorySegment if we want to perform operations in native (e.g.
>> SIMD, GPU processing). I'm sure most of Java API / third party libraries
>> use primitive array, not MemorySegment. So I think it is better if we
>> pin on-heap memory and leverage it in native.
>>
>> Of course I understand it might be some worse behaviors especially GC,
>> so pinning might not be recommended for all of Java developers. But I
>> believe pinning is welcomed from developers who experienced in native
>> (C/C++/assembly/GPGPU and so on) because they want to offload processing
>> which needs large memory (preprocessed in Java).
>>
>>
>> Thanks,
>>
>> Yasumasa
>>
>>
>> 2023-08-28 18:48 に Maurizio Cimadamore さんは書きました:
>> > On 28/08/2023 08:29, Yasumasa Suenaga wrote:
>> >> Hi all,
>> >>
>> >> I'm very interested in FFM, especially generate assembly code in Java
>> >> and calling them with FFM [1].
>> > Hi,
>> > I've seen your work a year or so ago and have been very impressed by it
>> > :-)
>> >>
>> >> I think one of the performance neck is MemorySegment because all of
>> >> on-heap regions should be copied into off-heap region (native segment)
>> >> when they are referred from foreign function. So I'm expecting to
>> >> implement sort of pinning operation for on-heap MemorySegment like a
>> >> JNI. I guess it is mentioned in FFM update in last month [2], however
>> >> "Pinning of heap segments" does not have any links - I guess nobody is
>> >> working for this yet. Do you have any updates for pinning?
>> >>
>> >> I've played them with both OpenJDK 22 with pinning support [3] and
>> >> ffmasm (hand-assembler for Java powered by FFM) [4]. I added pin/unpin
>> >> method into Unsafe, and they are called by HeapMemorySegmentImpl.
>> >> Finally I got about 16x performance gain compared to non-pinning code
>> >> [5] on my laptop.
>> >>
>> >> I guess FFM API reaches to goal step by step, but it is still a
>> >> preview in JDK 21. I hope that pinning feature is supported into JDK
>> >> 22 because I believe we can leverage FFM for more fields! I'm happy to
>> >> contribute/help to implement pinning feature if it needs.
>> >
>> > While there's no doubt that in some applications and use cases pinning
>> > provides a significant performance boost, there are some challenges:
>> >
>> > 1. scope of pinning: is pinning allowed on a per-native-call basis? Or
>> > is it something more general?
>> > 2. does the garbage collection support region-based pinning [1] ?
>> > 3. why is pinning needed in the first place?
>> >
>> > (1) and (2) are very much linked. Not all garbage collectors support
>> > fine-grained pinning mechanism. Which means that, in most of them, if
>> > you pin, you effectively block GC for the entire duration of the pin
>> > operation (GC locker mechanism). This is something that, as I'm sure
>> > you understand, is not very desirable. For this reason, it might be
>> > better to consider a pinning API which only pins for the duration of a
>> > native call (e.g. in the shape of an additional linker option). While
>> > FFM could support more complex pinning policies (e.g. pin a segment
>> > inside an Arena, so that segment is unpinned when the arena is
>> > closed), given the uneven support for fine-grained pinning across GCs,
>> > I'm not sure such a general API (which is similar to your "MS::pin"
>> > method) would be a good idea. We're doing some experiments for adding
>> > a new linker option which allows for pin heap segments to be pinned
>> > when calling a downcall method handle, we're not yet sure of its
>> > inclusion, but it would be something worth publishing somewhere (when
>> > ready) so that developers (like you) can play with it and provide
>> > feedback.
>> >
>> > Then there's (3). Most of the times, pinning is used in order to
>> > interact with native calls from public-facing APIs that are "stuck"
>> > using array syntax. That is, in order to be user friendly, such API
>> > work with arrays - but then a problem arises when trying to use the
>> > contents of the array off-heap. But what if the memory was off-heap to
>> > begin with? Then no memory copy would be required. I believe the
>> > biggest impediment for off-heap memory being used directly has to do
>> > with the fact that, for users, interacting with an `int[]` is
>> > significantly easier than interacting with a `MemorySegment`, or a
>> > `ByteBuffer`. But what if we could provide some mechanism to create an
>> > "array view" over an off-heap memory region? Now clients would be able
>> > to use the beloved `[]` syntax, even if memory access remained
>> > off-heap.
>> >
>> > While we don't have any concrete proposal on this latter point, we do
>> > believe that the topic of making memory segments (or byte buffer)
>> > easier to access for "legacy clients" is inextricably linked to the
>> > topic of pinning of heap memory.
>> >
>> > [1] - https://openjdk.org/jeps/423
>> >
>> >>
>> >>
>> >> Thanks,
>> >>
>> >> Yasumasa
>> >>
>> >>
>> >> [1] https://github.com/YaSuenag/ffmasm
>> <https://urldefense.com/v3/__https://github.com/YaSuenag/ffmasm__;!!ACWV5N9M2RV99hQ!IGKbapxVG_QMwdL236-U-1XDKQLgz1imop4dEpsa2CqyUkuhuLs7lk5LYrCbzu3nH6uT0YTrJVKY329M7mInwXvCTRxW$>
>> >> [2]
>> >> https://mail.openjdk.org/pipermail/panama-dev/2023-July/019510.html
>> >> [3]
>> >>
>> https://github.com/YaSuenag/jdk/commit/f0a9b3705b3ecdf3dbb6b80cac9d53456f08f967
>> <https://urldefense.com/v3/__https://github.com/YaSuenag/jdk/commit/f0a9b3705b3ecdf3dbb6b80cac9d53456f08f967__;!!ACWV5N9M2RV99hQ!IGKbapxVG_QMwdL236-U-1XDKQLgz1imop4dEpsa2CqyUkuhuLs7lk5LYrCbzu3nH6uT0YTrJVKY329M7mInwbe47VCY$>
>> >> [4]
>> >>
>> https://github.com/YaSuenag/ffmasm/commit/925608538b936db1b311ae84e12fa0252058b7f4
>> <https://urldefense.com/v3/__https://github.com/YaSuenag/ffmasm/commit/925608538b936db1b311ae84e12fa0252058b7f4__;!!ACWV5N9M2RV99hQ!IGKbapxVG_QMwdL236-U-1XDKQLgz1imop4dEpsa2CqyUkuhuLs7lk5LYrCbzu3nH6uT0YTrJVKY329M7mInwbUJ6qi5$>
>> >> [5]
>> >>
>> https://github.com/YaSuenag/ffmasm/blob/ffm-pinning/benchmarks/vectorapi/src/main/java/com/yasuenag/ffmasm/benchmark/vectorapi/VectorOpComparison.java
>> <https://urldefense.com/v3/__https://github.com/YaSuenag/ffmasm/blob/ffm-pinning/benchmarks/vectorapi/src/main/java/com/yasuenag/ffmasm/benchmark/vectorapi/VectorOpComparison.java__;!!ACWV5N9M2RV99hQ!IGKbapxVG_QMwdL236-U-1XDKQLgz1imop4dEpsa2CqyUkuhuLs7lk5LYrCbzu3nH6uT0YTrJVKY329M7mInwRLDJMg-$>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/panama-dev/attachments/20230828/5ebb020c/attachment-0001.htm>