Reading binary data

Thu Nov 7 10:26:30 UTC 2019

On 07/11/2019 01:14, Samuel Audet wrote:
> I'm not personally convinced that we need all those safety features. 
> We've been doing well with ByteBuffer/JNI without all that, haven't we?
ByteBuffer is safe. Unsafe isn't. They are two very different examples. 
And the audience of the Java platform is quite big too. While for some 
clients it might be ok to live with the occasional crash 'caused by JNI' 
code, other clients might not be ok with that - and if your application 
consists of 256 jar files in the classpath (or modules, for what it's 
worth) and any of them might randomly crash the VM because of an unsafe 
operation, well you can see the maintenance nightmare.
> As part of JavaCPP, I only allow users to change the address in 
> subclasses of Pointer, something casual users don't usually do, and 
> that has worked out well too. I don't remember anyone "accidentally" 
> creating a subclass...
If you do somtething like that in the Java SE API I can 100% guarantee 
some developer will do that - not only that, some will even go at 
certain length to repackage their own 'unsafe pointer APIs' on Maven 
that you can use in order to 'get rid of all these constraints'. At 
least this is the story we've seen so far with other APIs.
>
> In any case, I think Swift would be a good reference for this. They've 
> been very busy experimenting with everything related to this:
> https://developer.apple.com/documentation/swift/swift_standard_library/manual_memory_management 
>
> There are also many many blog posts from users explaining what they 
> felt were pros and cons of each approach and various versions of Swift...
>
> Now, you said previously that this API is almost complete, but from 
> what you just wrote now, it sounds pretty incomplete to me. What kind 
> of time frame do you have in mind?

Ah - the old good Samuel trick :-)

Please tell where this inference came from? You asked if it was possible 
to do XYZ - I replied, with pointer to the code, saying that it *is* 
possible; but, since it's a protected functionality (for a good 
reason!), you don't like it, and you call the whole thing incomplete?

I could make the same arguments to you saying that I think JavaCPP is 
incomplete because it allows anyone to forge pointers out of thin air. 
But I'm not. I know that JavaCPP is good at what it does and for the 
audience it targets - in the JDK, we just have different constraints 
that you have. If, in your world, respecting these constraints === being 
incomplete, well, so be it.

Maurizio

>
> Samuel
>
> On 11/6/19 8:10 PM, Maurizio Cimadamore wrote:
>>
>> On 06/11/2019 04:32, Samuel Audet wrote:
>>> Hi, Maurizio,
>>>
>>> We do need /something/ to replace sun.misc.Unsafe, so that's 
>>> welcome, but I'm concerned about it not actually being able to do 
>>> the job. If I understand correctly, it is not possible to create a 
>>> MemorySegment directly from an arbitrary long integer: We need to 
>>> work around that with Pointer. (Or did I miss a piece of the API 
>>> somewhere?) However, if I understand correctly, MemorySegment is 
>>> scheduled to be released before Pointer, which again, if I 
>>> understand correctly, might not be for a few more years. If that's 
>>> the case, what are users meant to do in meantime?
>>
>> It is possible to create a memory address from a long:
>>
>> http://hg.openjdk.java.net/panama/dev/file/0798bb838349/src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/unsafe/ForeignUnsafe.java#l75 
>>
>>
>> This is, currently (and for good reasons) an unsafe operation, so it 
>> is defined in a non-exported package which can be --add-exports(ed) 
>> if the user really need to do something like that. We also have ideas 
>> on how to allow operations such as these in a _safe_ way - that is, 
>> we could allow creation of 'random' addressed from long (including 
>> null), provided that we do not allow to safely dereference them. 
>> While this is not a 100% solution it would support NULL cleanly, as 
>> well as all cases where native API deal in opaque pointers.
>>
>> Here there's a bigger, and very interesting theme, of how should an 
>> API containing both _safe_ and _unsafe_ parts should be defined in a 
>> way that (i) allow programmers to do what they have to do w/o being 
>> too punitive and (ii) prevents unsafe portions of the API to be 
>> repackaged and re-exported to innocuous clients operating in _safe_ 
>> mode. This is an hard trade-off to make, and it also doesn't help 
>> that, so far, JNI has always been treated as a _safe_ player.
>>
>> I don't think the Pointer API has anything to create a Pointer from a 
>> long (unless you refer to the trick of writing a long in memory, and 
>> then reading it back as a Pointer).
>>
>> Maurizio
>>
>>>
>>> Samuel
>>>
>>> On 10/31/19 8:55 AM, Maurizio Cimadamore wrote:
>>>>
>>>> On 30/10/2019 20:29, Ty Young wrote:
>>>>> As an API user I would much rather have the APIs being integrated 
>>>>> with one another rather than completely separate APIs. Spending 
>>>>> hours trying to figure out how to read the data -only to use the 
>>>>> incorrect way that seems to return a valid value but in actuality 
>>>>> isn't- isn't going to be fun. 
>>>>
>>>> I think there is a misunderstanding here on what these API are 
>>>> meant to be.
>>>>
>>>> The memory access API is a low level API, it sits in the 
>>>> foreign-memaccess branch; that's what my talk (which you might have 
>>>> seen on youtube) speak about. It's meant to be a general off-heap 
>>>> API, with focus on safety and performances - kind of a replacement 
>>>> for Unsafe::get/putInt and friends.
>>>>
>>>> The Pointer API is what's available in the 'foreign' branch - and 
>>>> does what it says on the tin: it provides a Java abstraction which 
>>>> models a C source pointer type. The Pointer API is for language 
>>>> interop; the memory access API is lower level (in fact you can 
>>>> build the Pointer API on top of the memory access API).
>>>>
>>>> I'm sorry to hear your frustration in using the API (I assume the 
>>>> Pointer API?) but w/o understanding more of what you are trying to 
>>>> do, it's gonna be hard for us to offer any help or guidance.
>>>>
>>>> Cheers
>>>> Maurizio
>>>>
>