Java object as a void*

Wed Feb 15 15:12:36 UTC 2023

Another thing I'd like to add: while "turn a Java object into a pointer" 
sounds simple, in reality if you look at the implementation of JNI 
global handles, it is not that much different from the hash map 
approach, in the sense that objects are stored into some (global) data 
structure [1]. Of course, resolving objects in this storage is simpler, 
since a handle is a tagged pointer directly into this data structure. 
But, this global data structure is also a very general-purpose storage 
mechanism. Depending on the use case, another kind of data structure 
might be more suited.

So, I'd say that using a hash map + integer keys is not as much of a 
workaround as you think it is. Since Java objects are managed by a GC, 
in reality we'll always need some kind of map/store that the GC knows 
about, since it can not track every arbitrary pointer value floating 
around in memory or CPU registers.

Jorn

[1] : 
https://github.com/openjdk/jdk/blob/0c9658446d111ec944f06b7a8a4e3ae7bf53ee8d/src/hotspot/share/gc/shared/oopStorage.cpp#L389-L436

On 15/02/2023 15:03, Maurizio Cimadamore wrote:
>
>
> On 15/02/2023 07:34, Manuel Bleichenbacher wrote:
>> Hi Maurizio
>>
>> Thanks for the prompt answer.
>>
>> I can understand that you are reluctant to introduce a new concept 
>> for a use cases that isn't particularly frequent.
>>
>> From what I read about ScopedValues, I doubt they will be helpful. In 
>> many of the use cases I'm dealing with, the void* parameter is used 
>> in a cross-thread context and has a lifetime, which is not aligned 
>> with a function call or a similar scope. A thread-local variable is 
>> not helpful for my use cases. Doesn't this imply that ScopedValues 
>> won't help either?
>
> ScopedValues can be inherited across threads, but only if the threads 
> are constructed/managed by a StructuredTaskScope:
>
> https://openjdk.org/jeps/437
>
> Which might or might not be flexible enough for the particular use 
> cases you have in mind. But I wanted to make clear that some 
> well-behaved form of muli-threading is supported by ScopedValues.
>
>>
>> Regarding JNI handles and resurrection I don't have sufficient 
>> insight into the implementation to comment on this. Since it is 
>> already possible to bind an object to a method handle, create an 
>> upcall stub for it and passed it around in native code, it would seem 
>> to me that part of the challenges have already been solved.
>
> The main difference is that when you call an "upcall" there is always 
> a known class that acts as an execution context for the code that runs 
> in the upcall. If an upcall binds one or more objects, these objects 
> will belong to the same context as the resulting upcall method handle 
> itself, so no problem arises.
>
> Allowing ordinary Java objects to be turned into JNI handles which can 
> be stored inside native libraries opens up a scenario where an object 
> created in one context might "leak" into another context (this is 
> possible if two clients interact with the same underlying native 
> library). This situation is new, and would have to be taken care of.
>
> Cheers
> Maurizio
>
>
>
>>
>> Regards
>> Manuel
>>
>> Am Mi., 15. Feb. 2023 um 00:21 Uhr schrieb Maurizio Cimadamore 
>> <maurizio.cimadamore at oracle.com>:
>>
>>     Hi Manuel,
>>     in principle, it would be doable to expose an API which creates a
>>     JNI
>>     handle out of a Java object and exposes it as a MemorySegment so
>>     that it
>>     can be passed to native functions. One thing to consider is that
>>     if we
>>     started passing Java objects around to native code, then we'd
>>     also have
>>     to start worrying about the same objects being "resurrected" in
>>     unexpected places (e.g. created by one classloader, accessed by
>>     another). Some of these issues are the very reasons behind the
>>     classloader restrictions behind System::loadLibrary [1]. Perhaps
>>     here,
>>     since the API doing the access is a Java API, it might be
>>     possible to
>>     insert some kind of class loader check.
>>
>>     That said, one of the main reason we did not add any special
>>     support for
>>     JNI handles is that, in addition to the workarounds you mention,
>>     a new
>>     and more powerful workaround is/will be possible from Java 20:
>>     ScopedValues [2]. That is, you can set up a scoped value, and
>>     "bind" it
>>     before the upcall runs (e.g. before you call the corresponding
>>     downcall). The bound value would then be in scope (as a sort of
>>     implicit
>>     parameter) inside the upcall code as well. Retrieving a scoped
>>     value is
>>     also quite fast (compared e.g. to a ThreadLocal).
>>
>>     For this reason we'd like to see how using ScopedValues goes before
>>     thinking about adding more ad-hoc machinery to expose JNI handles
>>     in Panama.
>>
>>     Cheers
>>     Maurizio
>>
>>     [1] -
>>     https://docs.oracle.com/javase/7/docs/technotes/guides/jni/jni-12.html#libmanage
>>     [2] - https://openjdk.org/jeps/429
>>
>>
>>     On 14/02/2023 18:39, Manuel Bleichenbacher wrote:
>>     > Hi all
>>     >
>>     > Many native SDKs allow the caller to pass a void* or long as an
>>     > additional parameter. This parameter is then later provided as a
>>     > reference to the original call, either in the context of a
>>     callback
>>     > function call or as an attribute of an operating system object.
>>     The
>>     > native SDK will not interpret the parameter in any way, neither
>>     as a
>>     > number nor as a pointer. It's up to the caller to know what it
>>     represents.
>>     >
>>     > Examples are:
>>     >
>>     > lParam in Windows' SendMessage() -
>>     >
>>     https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-sendmessage
>>     <https://urldefense.com/v3/__https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-sendmessage__;!!ACWV5N9M2RV99hQ!ObuzTPS17OHPuAWdkoUdZ15H6ZrH5uj07MP5jV1Zccc4rS1XvpdILx3cG-IefKuy_8uY3YyZwU19dNZ7GxNigifnyTkGC6mZlA$>
>>     >
>>     > dwNewLong in Windows' SetClassLongPtr() -
>>     >
>>     https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-setclasslongptrw
>>     <https://urldefense.com/v3/__https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-setclasslongptrw__;!!ACWV5N9M2RV99hQ!ObuzTPS17OHPuAWdkoUdZ15H6ZrH5uj07MP5jV1Zccc4rS1XvpdILx3cG-IefKuy_8uY3YyZwU19dNZ7GxNigifnyTmGQRUeqw$>
>>     >
>>     > arg in POSIX's pthread_create() -
>>     >
>>     https://pubs.opengroup.org/onlinepubs/000095399/functions/pthread_create.html
>>     <https://urldefense.com/v3/__https://pubs.opengroup.org/onlinepubs/000095399/functions/pthread_create.html__;!!ACWV5N9M2RV99hQ!ObuzTPS17OHPuAWdkoUdZ15H6ZrH5uj07MP5jV1Zccc4rS1XvpdILx3cG-IefKuy_8uY3YyZwU19dNZ7GxNigifnyTm2gkc1kA$>
>>     >
>>     > refcon in macOS' WritePipeAsync() -
>>     >
>>     https://developer.apple.com/documentation/iokit/iousbinterfaceinterface800/1639539-writepipeasync
>>     <https://urldefense.com/v3/__https://developer.apple.com/documentation/iokit/iousbinterfaceinterface800/1639539-writepipeasync__;!!ACWV5N9M2RV99hQ!ObuzTPS17OHPuAWdkoUdZ15H6ZrH5uj07MP5jV1Zccc4rS1XvpdILx3cG-IefKuy_8uY3YyZwU19dNZ7GxNigifnyTnv2uJlmA$>
>>     >
>>     > Is there a way to use this from Java for Java objects, i.e. to
>>     pass a
>>     > Java object as a void* and later convert the void* back to the
>>     Java
>>     > object?
>>     >
>>     > Since the FFM API exposes void* as MemorySegments, it would
>>     probably
>>     > mean that a MemorySegment could be created encapsulating a
>>     reference
>>     > to a Java object, and that the Java object can later be
>>     retrieved from
>>     > this MemorySegment.
>>     >
>>     > I am aware of two workarounds:
>>     >
>>     > - In the context of a callback function, the Java callback
>>     function
>>     > can be extended with an additional parameter and then the Java
>>     object
>>     > can be bound to it. This function is then used to create the up
>>     call
>>     > stub. Since the callback function already contains the
>>     reference, the
>>     > void* parameter is no longer needed and can be passed
>>     > MemorySegment.NULL. In my context (I/O processing with several
>>     > thousand functions calls per second), it likely is rather
>>     inefficient
>>     > as several thousand upcall stubs will need to be generated by
>>     second.
>>     >
>>     > - Instead of the Java object, a generated integer key could be
>>     passed
>>     > as the void*. The link between the key and the Java object
>>     would need
>>     > to be managed in a hash map. Possible, but more of a workaround.
>>     >
>>     > Regards
>>     > Manuel
>>     >
>>     >
>>     >
>>     >
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/panama-dev/attachments/20230215/72df8a72/attachment-0001.htm>