Feedback about an experiment to embed Python interpreter with FFI API

[Maurizio Cimadamore]

Hi Maxim
> The topic I want to talk about is a small flag `--enable-native-access=X`
> that looks harmless and useful at first glance. tl,dr: it is neither
> harmless nor useful. The reasoning that "user must opt-in to use unsafe
> APIs" implies that: (1) unsafe APIs are something illegal and should be
> avoided; (2) user has competencies to make such decisions; (3) user has
> other choice than use unsafe APIs. From my point of view, all three
> assumptions are false.

This has been discussed before (see [1]). My main suggestion here is to 
avoid over-generalization - e.g. speaking about "programs" and "users" 
in general is almost always incorrect and prone to simplifications and 
biases. Which programs? Which users? Note that there are actual 
instances of users who found the flag very useful (see [2]), which 
suggests that reality is probably more fragmented than what we'd like it 
to be.

Native access is not illegal, of course (though some of the wording 
around the exceptions thrown when the flag isn't there might be 
unfortunate and suggest that). But doing native access (of any kind) 
puts the application under much greater risk of misuse, as there is no 
way for the JVM to enforce safety for native code (e.g. module access 
boundaries). The fact that other mechanisms such as System.exit, or 
process builder exists by which bad things can happen and for which no 
protection exists is not, in itself, a justification as to why accessing 
native features of the Java API should not be protected.

When a developer uses JNI a number of things have to happen for that 
code to be in a workable state. When writing a program, native code has 
to be compiled - when executing, native libraries must be made 
available. These things all require command line flags (e.g. 
-Djava.library.path). There are, of course, frameworks that will 
side-step the library loading requirement by embedding the desired 
library somewhere in a dependency jar file, extract it into a temp 
folder, and then load the library directly from there, so that these 
libraries appear to work "seamlessly".

With the foreign API, when writing code there's no requirement to use 
GCC/clang. So that's one obstacle removed for the developer. And, when 
executing, assuming you are only using system libraries (e.g. posix, 
windows API), an application could just start calling native functions 
seamlessly (w/o any kind of workaround, unlike in JNI).

This means that it is now much easier for seemingly innocuous Java code 
to behave in unpredictable ways, hence the flag.


>
>
> The constructive part: what will be more helpful and much less 
> burdensome? I
> would say that much better alternative to a feature flag is just to 
> record
> all modules (or classes) which do native access, either by FFI or (in
> future) by JNI. Then just include list of such modules or classes in 
> crash
> dump. Straightforward: the VM has crashed, but classes A, B or C might be
> an issue since they used native access. Try to removing these classes 
> first
> before reporting any bugs. No deployment burden, no breaking of 
> compatibilty
> by introducing new restrictions and much more helpful crash dumps.


I think a better way, which we have thought about (but have not got to 
yet) is to think about native access as a permission that can be granted 
to a module to other modules. Let's say you have an application module 
App - and App depends on Foo and Bar, where Bar requires native access. 
If the required information is captured in the module system, then the 
application developer can simply grant native access to App, and App 
will be responsible to transfer the native access permission as required 
to the modules which requires it. I think such a solution would address 
most of the issues with the current flag (e.g. finding all the modules 
which need native access and granting them explicitly) and retain the 
spirit of capturing important permission information at the module-graph 
level (so that, if an application is executed w/o enough permission, it 
would fail to start early, rather than wait for the first unsafe method 
to be called).

Another option we considered was to turn native access errors into 
warnings if no --enable-native-access flag is specified (but retain 
existing behavior if the flag is specified). This would allow for a 
smoother migration, while allowing clients that want to be strict about 
native access (see [2]) to still function in the way they do today. To 
conclude, I think there are possible tweaks to make the flag more useful 
w/o merely turning it into a debugging feature as you propose.

Maurizio

[1] - 
https://mail.openjdk.java.net/pipermail/panama-dev/2021-September/015036.html
[2] - 
https://mail.openjdk.java.net/pipermail/panama-dev/2021-December/015981.html