Reading binary data

[Maurizio Cimadamore]

On 07/11/2019 11:38, Ty Young wrote:
>
> On 11/6/19 5:10 AM, Maurizio Cimadamore wrote:
>> It is possible to create a memory address from a long:
>>
>> http://hg.openjdk.java.net/panama/dev/file/0798bb838349/src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/unsafe/ForeignUnsafe.java#l75 
>>
>>
>> This is, currently (and for good reasons) an unsafe operation, so it 
>> is defined in a non-exported package which can be --add-exports(ed) 
>> if the user really need to do something like that. We also have ideas 
>> on how to allow operations such as these in a _safe_ way - that is, 
>> we could allow creation of 'random' addressed from long (including 
>> null), provided that we do not allow to safely dereference them. 
>> While this is not a 100% solution it would support NULL cleanly, as 
>> well as all cases where native API deal in opaque pointers.
>>
>> Here there's a bigger, and very interesting theme, of how should an 
>> API containing both _safe_ and _unsafe_ parts should be defined in a 
>> way that (i) allow programmers to do what they have to do w/o being 
>> too punitive and (ii) prevents unsafe portions of the API to be 
>> repackaged and re-exported to innocuous clients operating in _safe_ 
>> mode. This is an hard trade-off to make, and it also doesn't help 
>> that, so far, JNI has always been treated as a _safe_ player.
>
>
> Speaking purely from an API design perspective, having the "safe"(high 
> level) API be an extension of the "unsafe"(low level) API and hiding 
> the lower level functionality via the protected visibility modifier, 
> as was suggested by Samuel, and/or packaging separation sounds like 
> the only real solution. Safety as an extension, if you will.
Sorry - in the Java SE API, safety is never 'an extension'. You might 
not like it, but you will have to live with it.
>
>
> Regarding the topic of safety, it feels like important context is 
> being forgotten... or maybe I'm missing something. C/C++ is primarily 
> used for sensitive, low level, and hardware level programming. Any bad 
> or incorrect API usage(integer overflows, for example) could result in 
> the entire system -including the JVM- to explode and crash. The "safe" 
> Pointer API, even with all of its safety, isn't going to prevent 
> system crashes from C/C++ code.

> Heck, even within Java itself without Panama, it's possible to cause 
> system chaos because *all roads eventually lead to Rome*. Anything 
> that affects the system can affect the JVM and code executed in the 
> JVM can negatively affect the system.

If you have examples of something that doesn't do any JNI whatsoever, 
and yet leads to a VM crash we'd like to know, do you have any examples 
in mind? (we typically treat these cases as VM or JDK bugs).

>
>
> Which is why I was confused by both the presentation and now(regarding 
> your newest email) in regards to Panama preventing crashing the VM 
> directly or by extension. How is such a thing even conceivably 
> possible anymore than preventing an ordinary bug from within pure Java 
> code and crashing the system that way? Ignoring the context of Panama, 
> literally any buggy code from any library at any point could cause 
> huge failure.
The memory access API (leave aside the Pointer API for now) is an API 
for accessing off-heap memory. I think both you and Samuel are making 
the inference that off-heap means "native". But that doesn't have to be 
the case. There are plenty of use cases in the wild where developers 
just want to be able to deserialize an object graph off-heap, mapping it 
only a file, etc. These use cases have nothing inherently unsafe to 
them. And there's a lot of them, only you don't see them because right 
now they are catered by the ByteBuffer API (which does a so and so job 
at that), or some ad-hoc APIs such as Netty's ByteBuf or ChronicleBytes.
>
>
> Not saying "safety" doesn't matter, from a high level use case it 
> absolutely does and is important, just that hiding API functionality 
> that people want/need for this specific reason doesn't make a whole 
> lot of sense. Panama can't conceivable be a one API army that somehow 
> either fixes everyone's buggy code, prevents them from writing it to 
> begin with, and/or smooths over buggy code. Nothing really can.

(I'll skip over the "functionality that people want/need" - these 
comments are baseless, unless you claim to have some telepathic 
connection with all of the varied Java ecosystem - I sometimes wish I 
had that!)

So, are you saying that adding an extra compiler flag when compiling 
your program that requires privileged operation is 'too much' ? Honestly 
this discussion seems to be going in circles. The memory access API has 
been out there for almost 6 months now - how much code have you written 
with it (same for Samuel) ? Are you sure that what you now describe as a 
major problem isn't, in reality, something that will only be required by 
people requiring low level access to write frameworks like JavaCPP 
itself and JNR? If so, are you sure that _everybody_ will care, as you 
claim they will? Or are you mistaking the 10% case with the 90% case?

Suggestion, write a lot of code using the memory access API, report back 
your experience and the chances your feedback will be taken more 
seriously will increase significantly (cross my heart!).

Maurizio

>
>
>>
>> I don't think the Pointer API has anything to create a Pointer from a 
>> long (unless you refer to the trick of writing a long in memory, and 
>> then reading it back as a Pointer).
>>
>> Maurizio
>>