Reading binary data

[Samuel Audet]

I'm not personally convinced that we need all those safety features. 
We've been doing well with ByteBuffer/JNI without all that, haven't we?
As part of JavaCPP, I only allow users to change the address in 
subclasses of Pointer, something casual users don't usually do, and that 
has worked out well too. I don't remember anyone "accidentally" creating 
a subclass...

In any case, I think Swift would be a good reference for this. They've 
been very busy experimenting with everything related to this:
https://developer.apple.com/documentation/swift/swift_standard_library/manual_memory_management
There are also many many blog posts from users explaining what they felt 
were pros and cons of each approach and various versions of Swift...

Now, you said previously that this API is almost complete, but from what 
you just wrote now, it sounds pretty incomplete to me. What kind of time 
frame do you have in mind?

Samuel

On 11/6/19 8:10 PM, Maurizio Cimadamore wrote:
> 
> On 06/11/2019 04:32, Samuel Audet wrote:
>> Hi, Maurizio,
>>
>> We do need /something/ to replace sun.misc.Unsafe, so that's welcome, 
>> but I'm concerned about it not actually being able to do the job. If I 
>> understand correctly, it is not possible to create a MemorySegment 
>> directly from an arbitrary long integer: We need to work around that 
>> with Pointer. (Or did I miss a piece of the API somewhere?) However, 
>> if I understand correctly, MemorySegment is scheduled to be released 
>> before Pointer, which again, if I understand correctly, might not be 
>> for a few more years. If that's the case, what are users meant to do 
>> in meantime?
> 
> It is possible to create a memory address from a long:
> 
> http://hg.openjdk.java.net/panama/dev/file/0798bb838349/src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/unsafe/ForeignUnsafe.java#l75 
> 
> 
> This is, currently (and for good reasons) an unsafe operation, so it is 
> defined in a non-exported package which can be --add-exports(ed) if the 
> user really need to do something like that. We also have ideas on how to 
> allow operations such as these in a _safe_ way - that is, we could allow 
> creation of 'random' addressed from long (including null), provided that 
> we do not allow to safely dereference them. While this is not a 100% 
> solution it would support NULL cleanly, as well as all cases where 
> native API deal in opaque pointers.
> 
> Here there's a bigger, and very interesting theme, of how should an API 
> containing both _safe_ and _unsafe_ parts should be defined in a way 
> that (i) allow programmers to do what they have to do w/o being too 
> punitive and (ii) prevents unsafe portions of the API to be repackaged 
> and re-exported to innocuous clients operating in _safe_ mode. This is 
> an hard trade-off to make, and it also doesn't help that, so far, JNI 
> has always been treated as a _safe_ player.
> 
> I don't think the Pointer API has anything to create a Pointer from a 
> long (unless you refer to the trick of writing a long in memory, and 
> then reading it back as a Pointer).
> 
> Maurizio
> 
>>
>> Samuel
>>
>> On 10/31/19 8:55 AM, Maurizio Cimadamore wrote:
>>>
>>> On 30/10/2019 20:29, Ty Young wrote:
>>>> As an API user I would much rather have the APIs being integrated 
>>>> with one another rather than completely separate APIs. Spending 
>>>> hours trying to figure out how to read the data -only to use the 
>>>> incorrect way that seems to return a valid value but in actuality 
>>>> isn't- isn't going to be fun. 
>>>
>>> I think there is a misunderstanding here on what these API are meant 
>>> to be.
>>>
>>> The memory access API is a low level API, it sits in the 
>>> foreign-memaccess branch; that's what my talk (which you might have 
>>> seen on youtube) speak about. It's meant to be a general off-heap 
>>> API, with focus on safety and performances - kind of a replacement 
>>> for Unsafe::get/putInt and friends.
>>>
>>> The Pointer API is what's available in the 'foreign' branch - and 
>>> does what it says on the tin: it provides a Java abstraction which 
>>> models a C source pointer type. The Pointer API is for language 
>>> interop; the memory access API is lower level (in fact you can build 
>>> the Pointer API on top of the memory access API).
>>>
>>> I'm sorry to hear your frustration in using the API (I assume the 
>>> Pointer API?) but w/o understanding more of what you are trying to 
>>> do, it's gonna be hard for us to offer any help or guidance.
>>>
>>> Cheers
>>> Maurizio
>>>