Reading binary data

[Ty Young]

On 11/7/19 6:58 AM, Maurizio Cimadamore wrote:
>
> On 07/11/2019 11:38, Ty Young wrote:
>>
>> On 11/6/19 5:10 AM, Maurizio Cimadamore wrote:
>>> It is possible to create a memory address from a long:
>>>
>>> http://hg.openjdk.java.net/panama/dev/file/0798bb838349/src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/unsafe/ForeignUnsafe.java#l75 
>>>
>>>
>>> This is, currently (and for good reasons) an unsafe operation, so it 
>>> is defined in a non-exported package which can be --add-exports(ed) 
>>> if the user really need to do something like that. We also have 
>>> ideas on how to allow operations such as these in a _safe_ way - 
>>> that is, we could allow creation of 'random' addressed from long 
>>> (including null), provided that we do not allow to safely 
>>> dereference them. While this is not a 100% solution it would support 
>>> NULL cleanly, as well as all cases where native API deal in opaque 
>>> pointers.
>>>
>>> Here there's a bigger, and very interesting theme, of how should an 
>>> API containing both _safe_ and _unsafe_ parts should be defined in a 
>>> way that (i) allow programmers to do what they have to do w/o being 
>>> too punitive and (ii) prevents unsafe portions of the API to be 
>>> repackaged and re-exported to innocuous clients operating in _safe_ 
>>> mode. This is an hard trade-off to make, and it also doesn't help 
>>> that, so far, JNI has always been treated as a _safe_ player.
>>
>>
>> Speaking purely from an API design perspective, having the 
>> "safe"(high level) API be an extension of the "unsafe"(low level) API 
>> and hiding the lower level functionality via the protected visibility 
>> modifier, as was suggested by Samuel, and/or packaging separation 
>> sounds like the only real solution. Safety as an extension, if you will.
> Sorry - in the Java SE API, safety is never 'an extension'. You might 
> not like it, but you will have to live with it.


Going by the Properties class, It doesn't look like all the JDK 
developers got the memo on that.


>
>>
>>
>> Regarding the topic of safety, it feels like important context is 
>> being forgotten... or maybe I'm missing something. C/C++ is primarily 
>> used for sensitive, low level, and hardware level programming. Any 
>> bad or incorrect API usage(integer overflows, for example) could 
>> result in the entire system -including the JVM- to explode and crash. 
>> The "safe" Pointer API, even with all of its safety, isn't going to 
>> prevent system crashes from C/C++ code.
>
>> Heck, even within Java itself without Panama, it's possible to cause 
>> system chaos because *all roads eventually lead to Rome*. Anything 
>> that affects the system can affect the JVM and code executed in the 
>> JVM can negatively affect the system.
>
> If you have examples of something that doesn't do any JNI whatsoever, 
> and yet leads to a VM crash we'd like to know, do you have any 
> examples in mind? (we typically treat these cases as VM or JDK bugs).


I never said crash specifically, only that it can cause chaos. The only 
thing I'm aware of on the top of my head is attempting to create an 
insane Integer.MAX amount of threads and silently bugging the entire 
system. Never heard of anyone doing it but I know it's possible.


>
>>
>>
>> Which is why I was confused by both the presentation and 
>> now(regarding your newest email) in regards to Panama preventing 
>> crashing the VM directly or by extension. How is such a thing even 
>> conceivably possible anymore than preventing an ordinary bug from 
>> within pure Java code and crashing the system that way? Ignoring the 
>> context of Panama, literally any buggy code from any library at any 
>> point could cause huge failure.
> The memory access API (leave aside the Pointer API for now) is an API 
> for accessing off-heap memory. I think both you and Samuel are making 
> the inference that off-heap means "native". But that doesn't have to 
> be the case. There are plenty of use cases in the wild where 
> developers just want to be able to deserialize an object graph 
> off-heap, mapping it only a file, etc. These use cases have nothing 
> inherently unsafe to them. And there's a lot of them, only you don't 
> see them because right now they are catered by the ByteBuffer API 
> (which does a so and so job at that), or some ad-hoc APIs such as 
> Netty's ByteBuf or ChronicleBytes.
>>
>>
>> Not saying "safety" doesn't matter, from a high level use case it 
>> absolutely does and is important, just that hiding API functionality 
>> that people want/need for this specific reason doesn't make a whole 
>> lot of sense. Panama can't conceivable be a one API army that somehow 
>> either fixes everyone's buggy code, prevents them from writing it to 
>> begin with, and/or smooths over buggy code. Nothing really can.
>
> (I'll skip over the "functionality that people want/need" - these 
> comments are baseless, unless you claim to have some telepathic 
> connection with all of the varied Java ecosystem - I sometimes wish I 
> had that!)


Going by Samuel's remark on the API not being finished it seems so.


>
> So, are you saying that adding an extra compiler flag when compiling 
> your program that requires privileged operation is 'too much' ?


I'm not entirely sure what point is trying to be made. If it's something 
that can be done without then it's best to avoid it, of course. If it 
isn't... well, necessary evil.


> Honestly this discussion seems to be going in circles.


Your two points about the recurring theme was in itself a circle. Hiding 
functionality that someone might want is a burden on that person. Ideas 
were thrown around that may or may not, given context, make sense but at 
least it's *something*.


> The memory access API has been out there for almost 6 months now - how 
> much code have you written with it (same for Samuel) ?


I had an idea of what I wanted to do with it but given that I need both 
Pointer and memoryaccess APIs I kind of can't nor am I entirely sure 
that it provides what I need. Is it possible to get an exported function 
like getProcAddress does and refer to that address as a function? Is 
there a source branch that contains both?


> Are you sure that what you now describe as a major problem isn't, in 
> reality, something that will only be required by people requiring low 
> level access to write frameworks like JavaCPP itself and JNR?


Memory access is in itself a low level API, so yes? I get the point 
being made but it's still a use case...


> If so, are you sure that _everybody_ will care, as you claim they 
> will? Or are you mistaking the 10% case with the 90% case?


Never said such a thing.  And why divide things up like that when you 
could cover 100% of the use cases? Panama has an opportunity to provide 
functionality that Java hasn't ever been able to offer in any easily 
accessible fashion. Why not offer more if possible for those that need it?


>
> Suggestion, write a lot of code using the memory access API, report 
> back your experience and the chances your feedback will be taken more 
> seriously will increase significantly (cross my heart!).
>
> Maurizio
>
>>
>>
>>>
>>> I don't think the Pointer API has anything to create a Pointer from 
>>> a long (unless you refer to the trick of writing a long in memory, 
>>> and then reading it back as a Pointer).
>>>
>>> Maurizio
>>>