some thoughts on panama/jextract

[Jorn Vernee]

We're trying to make an API that allows you to access and use native 
libraries, and we intend to support any use case that is needed.

This _includes_ being able to dereference pointers that a user gets back 
from a native library, even though we can not be sure that this pointer 
is safe to dereference. But, here we have a problem; allowing users to 
dereference arbitrary memory also allows users to hard-crash the VM, 
i.e. generating an hs_err_pidXXXX.log file and spitting out this message:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ffaf5dae8ba, 
pid=21104, tid=24208
#
# JRE version: OpenJDK Runtime Environment (15.0) (fastdebug build 
15-internal+0-adhoc.Jorn.foreign-abi)
# Java VM: OpenJDK 64-Bit Server VM (fastdebug 
15-internal+0-adhoc.Jorn.foreign-abi, mixed mode, sharing, tiered, 
compressed oops, g1 gc, windows-amd64)
# Problematic frame:
# V  [jvm.dll+0xc7e8ba]  Unsafe_GetByte+0x1ca
#
# Core dump will be written. Default location: 
C:\projects\memaccess\hs_err_pid21104.mdmp
#
# An error report file with more information is saved as:
# C:\projects\memaccess\hs_err_pid21104.log
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#

This is quite different from the normal Java behavior of throwing an 
exception with a neat stack trace, where you can even inspect the 
running application at that point with a debugger. We believe that such 
a behavior change should be an explicit opt-in. This makes it for 
instance easier to find the source of bugs when such a crash happens, 
since the places where such bugs originate are all 'marked' in the code.

"But you can already do this by calling native code!" Yes, but in that 
case you have left the safety of Java, i.e. you are already explicitly 
opting in to potential VM crashes.

And, following the philosophy of the module system; if I write a library 
where I need to opt-into something, any clients of my library should 
also op-in. This is where the command line flag comes in, to be replaced 
by a more targeted mechanism and extended to SystemABI as well.

This whole thing is not about punishing certain native libraries, but 
we're thinking about this from a Java perspective, and for Java, being 
able to dereference arbitrary memory (and potentially hard-crash the VM) 
is a big change, that should be handled with care.

Jorn

On 13/01/2020 01:53, Michael Zucchi wrote:
> On 10/1/20 9:10 pm, Maurizio Cimadamore wrote:
>>
>> The point I'm making is about what the default should be. I think 
>> there's no question that, at the expressiveness level, you can make 
>> things to work in the current scheme. The only real question here is 
>> one: should pointers returned by native libraries be de-referenceable 
>> by default or not? 
>
> I don't see how that's the only real question, and honestly I don't 
> see how it's even a question in the first place.  C support simply 
> requires both and which is used in any case is *explicitly defined by 
> the API you're accessing*. Thus in the context of "calling C 
> functions", having one way or another as a "default" doesn't have any 
> meaning because there is simply no such default in C.  C relies on 
> both being available to be able to efficiently do useful work and 
> there are technical trade-offs when using either approach.
>
> Therefore any 'default' is not technical but political and the purpose 
> seems to be to:
>
> 1. Inform the binding-developer they have to be extra careful and that
>    they're using a 'bad' library (according to some specific but
>    ultimately arbitrary definition of 'bad'); and
> 2. Inform the binding-user-developer they are using something
>    potentially dangerous or 'unsafe'.
>
> But 1 is mostly meaningless and anyway redundant - and frankly 
> insulting, and 2 is actively misleading as i demonstrated with the 
> memcpy example.
>
> And you keep using loaded language like 'ill behaved' for one of the 
> cases.  But this title is simply not true at face value in a technical 
> sense and comes across as politically charged passive-aggressive 
> framing.  Such language is neither helpful nor necessary.
>
> I have less time to look at this now i'm back at work but will 
> continue my experiments* and report on them, and will follow the 
> project to see what you guys come up with.  From past experience i'm 
> pretty confident you'll come up with a decent solution in the end.  
> But it's interesting to see the process in motion and perhaps 
> contribute to its shaping rather than just passively consume the 
> results as I typically do.
>
> Regards,
>  Michael
>
> [*] This is primarily a deep deep 30+ year hobby for me that i've 
> basically dedicated/wasted my life to/on.  While I use most things I 
> learn for work and I learnt a great deal from my hobby in order to do 
> that better, my personal driver is honing the craft of turning a cpu 
> into an efficient and useful tool.  I have a lot of free time that I 
> often spend on all sorts of obtuse details, it's my main source of 
> entertainment and its typically about the journey and not the 
> destination.
>
> But debating is definitely not one of the things I enjoy spending my 
> time on!
>