JEP proposed to target JDK 22: 454: Foreign Function & Memory API

Mon Oct 9 15:51:41 UTC 2023

Hi Rony,
I believe you have made your position clear a number of time, both on 
panama-dev and on jdk-dev.

It is fair to say that, while we are not dismissing your concerns, our 
assessment of the overal impact of the new mechanism on the wider 
ecosystem is different than yours. You do not believe that the 
application packager is the right person to grant access to native code 
being used by the application. This has been discussed at length [2] and 
it seems that past attempts to convince you of the contrary were all 
unsuccessful. So I'd rather not rehash the same discussion here, and 
keep this discussion focussed on FFM (which is the subject of this thread).

The restricted method mechanism in FFM has been available for a long 
time - first time I see the mechanism being described is here:

https://openjdk.org/jeps/412

(from a couple of years ago).

Note that in Java 17 there were no warnings issued for FFM, just errors. 
Following some feedback [1] which discussed many of the topics also more 
recently touched in the JNI discussion, we have decided to issue 
warnings for usage of restricted methods, as a transitionary measure. 
The reasons for doing that were:

* there was no way to enable native access for executable jars (we are 
adding that now)
* there was no way to enable native access for modules defined 
programmatically (we have added that in Java 20)
* there were (as for JNI) concerns involving the lack of a mechanism to 
propagate the native access permissions from a module to its dependencies

After which, no further feedback has been received. To be clear, we 
believe that the above issues (and few others, such as adquate javadoc 
support, and availability of static analyzers) are still very important, 
and we do not plan to turn the warnings into errors before all the above 
issues are addressed in a way that is satisfactory.

You ask:

> Please pardon: what has JNI to do with FFM? 

I think there are a lot of things that JNI and FFM have in common, and 
there's also some things they do NOT have in common. For instance, JNI 
is much more unsafe, because, using its JNI functions, it can access any 
fields or methods bypassing the access checks that would normally be 
enforced by the Java runtime.

But both JNI and FFM compromise memory safety - e.g. they can create a 
situation where a client of a ByteBuffer, or a MemorySegment crashes 
because the region of memory behind the byte buffer/memory segment is no 
longer there. The way they do this is different (JNI does that using 
another JNI function, while FFM does that using restricted methods, such 
as MemorySegment::reinterpret). Also, both JNI and FFM can trigger 
library loading which can itself (via library loading hook) cause yet 
more native code to be executed.

In the current world, there is _no way_ for applications (e.g. pure Java 
ones) to guarantee that memory safety is not going to be compromised by 
any of their 3rd, 4th or 5th party dependencies. We would like to change 
that. The mechanism by which native access is granted, while consistent 
with similar flags (e.g. `--illegal-access=permit`, or, more recently 
`-XX:+EnableDynamicAgentLoading`) might not be perfect, and we we are 
committed to make it better (see above). But we strongly believe in a 
world where native access requires an opt-in.

[1] - 
https://mail.openjdk.org/pipermail/panama-dev/2021-September/014990.html
[2] - https://mail.openjdk.org/pipermail/jdk-dev/2023-August/008157.html

On 09/10/2023 15:16, Rony G. Flatscher wrote:
> It seems that the responsible people for this JEP have not taken the 
> objections into account that have been brought forward here in this 
> group (jdk-dev) and the panama-group (panama-dev), unless I am 
> misreading the JEP (the same goes for the idea in this context to do 
> the same all of a sudden with JNI usages 
> <https://openjdk.org/jeps/8307341>). Interestingly, there is no 
> warning planned if not acknowledging that Java/OpenJDK itself uses 
> native code.
>
> After giving samples in the panama-dev list and no more communication 
> took place after that there I assumed that the problem got understood.
>
> It seems that that was not the case and I am wondering why and who is 
> responsible, also where any related discussions took place (which 
> mailing lists), such that it becomes possible to read up and to learn 
> the assumptions and the arguments discussed there.
>
> ---
>
> For the record: following your advice further communication took place 
> in the panama-dev group starting with the posting at 
> <https://mail.openjdk.org/pipermail/panama-dev/2023-September/019869.html>. 
>
>
> As one of the problems in the various communications seems to have 
> been that some of the developers who came up with the idea to issue 
> such a warning addressed to /application packagers/ do not realize the 
> huge deployments of Java/OpenJDK as a shared, system wide Java runtime 
> environment (JRE) in which its Java class libraries get directly 
> exploited by stand-alone Java programs (and not packaged 
> applications). In the JRE context there are no "application packagers" 
> who could control how the JVM gets configured at package creation time 
> as there are no "application packages" (the JEP only looks at 
> executable jar files and application packages which may have their 
> private Java runtime environment coming with them, it does not take 
> into account the more important use case, namely having the 
> Java/OpenJDK runtime installed system wide and exploiting it as a 
> common system resource).
>
> Therefore the warning will be shown to the wrong (!) people becoming a 
> useless but frightening warning, surprising those who have been using 
> Java programs without any problems and confidence for years, and now 
> all of a sudden will be scared out of the blue using those very same 
> Java programs if these happen to exploit native functionality. Regular 
> users of Java programs will not be able to understand why and how to 
> remedy the situation. Rather they will be scared away from using Java 
> programs caused by an unjustified scary warning issued by Java itself.
>
> System wide JRE deployments are quite common as they offer many 
> benefits, among them the saving of a lot of resources (by not having 
> to distribute separate Java/OpenJDK environments for each single 
> program individually), another one being able to apply security 
> updates globally to a system wide installed Java runtime environment 
> and thereby fixing it for all Java programs that employ/use it. There 
> are more benefits like being able to switch the JREs to check out 
> different versions of Java/OpenJDK via JAVA_HOME for Java programs, 
> making the handling of the runtime much easier, less cumbersome, etc.
>
> To point this out the e-mail thread starting with 
> <https://mail.openjdk.org/pipermail/panama-dev/2023-September/019894.html> 
> tried to communicate this in the panama-dev list followed by three 
> short samples in which context this would create - even unfixable - 
> problems (just click "Next message (by thread)" at the top or the 
> bottom) for the Java users. As there have been no follow-ups to any of 
> these, I thought that the problem was understood but it seems that 
> they just got ignored instead.
>
> The last two samples related to the scripting framework and to JDBC 
> drivers and actually should of course have highlighted the fact that 
> any application of SPI (service provider interface) will make it 
> practically impossible (even for application packagers/developers at 
> packaging time) to know in advance which service implementations will 
> get deployed at runtime. Maybe it was not as evident as intended hence 
> pointing out SPI explicitly now.
>
> The SQLite JDBC driver sample in addition had the intention to demo a 
> real world jar that includes the native database management code for 
> all the major operating systems by design, yet, the current "warning 
> proposal" would not allow the SQLite jar authors to inhibit that 
> dreadful warning (only if it were a launched executable jar which it 
> will never be).
>
> ---
>
> So - after re-reading the current version of the JEP - it may be the 
> case that I misunderstand the current formulation in this context.
>
> If not, I would strongly object. The reason being in a nutshell that 
> regular users of Java programs (and Java programs using Java class 
> libraries that employ JNI like SQLite which includes SQLite in native 
> code for the most important operating systems) will be told that their 
> deployed Java programs have become dangerous to use (all of a sudden) 
> starting with version 22, and later versions of Java/OpenJDK will even 
> intentionally kill such deployed Java programs by issuing an 
> artificial error without a good reason.
>
> With other words, the preoccupation is that you are about to (probably 
> inadvertently) damage the perception of a great and safe Java 
> ecosystem for good by frightening the users of the Java ecosystem with 
> uncalled for safety warnings and later even harming (!) them if they 
> dare to continue to use a perfectly safe Java ecosystem outside of 
> packaged Java applications (add to that that Java/OpenJDK itself must 
> employ stable native code for good reasons, and safely so). 
> [Perception at one time already caused the industry to pull support 
> for Java in a strategic deployment field (browser plugins).]
>
> Actually, thinking about this warning proposal, it should be pulled 
> altogether in its current form. It does not increase the safety of 
> Java programs at all! If a native interaction causes a crash, that is 
> a "statement" that cannot be ignored (and not inhibited). That is also 
> the purpose of a crash: make it visible and inhibit any execution for 
> security reasons. Because of this any native code gets tested very 
> thoroughly. (Compare this to plain - "native clean" - Java code where 
> e.g. by mistake wrong radiation recipes could get created that could 
> kill people as a result.)
>
> Instead you want to warn *application packagers/developers* that they 
> are using jars that employ native access unless they indicate that 
> they know by configuring the JVM with a special new switch. This 
> sounds harmless at first sight for application packagers/developers. 
> However, if they dare not to do so then in the future their 
> applications will even get killed by the JVM, which is not harmless 
> anymore it is about forcing application packagers/developers. This 
> definitely does not sound like a constructive approach at all.
>
> What I would expect from the responsible persons, who all are 
> technical experts, to come up with a more constructive and more 
> appropriate solution that will make the targeted application 
> packagers/developers only aware by issuing such warnings with the 
> tools these application developers would be using like jlink, jar (for 
> executable jars) and maybe even javac, but not harm those who use Java 
> programs as they would not use those tools at all. Ideally you could 
> come up with a proper little developer's tool that would analyze 
> modules, application packages and jars and report which ones would 
> employ FFM (and JNI) calls. Do not bother the normal Java users and 
> let them safely keep on using their tested Java programs as they have 
> been doing successfully for the past years.
>
> ---rony
>
> P.S.: The FFM JEP-454 states: "To ensure a consistent approach to how 
> Java code interacts with native code, a related JEP proposes to 
> restrict the use of JNI in a similar way." Please pardon: what has JNI 
> to do with FFM?
>
> FFM allows normal Java programmers to inadvertently cause a crash of 
> the JVM which they may not be aware of (really, who would believe 
> that?) such that they should get warned? JNI does not allow a normal 
> Java programmer to inadvertently crash the JVM.
>
> Definitely JNI authors have been aware of crashes if the native code 
> is erroneous from day one after starting with their JNI development! 
> No Java programmer could crash the JVM using JNI inadvertently. 
> Professional JNI implementations get thoroughly tested and asserted 
> against crashes from day one, if they crash they do not get used, 
> simple as that! Those who write JNI know what they are doing and what 
> is dangerous and how to safe-guard. There is definitely no need to 
> warn JNI authors against potential crashes or Java programmers who use 
> a JNI library! They all know what they are doing.
>
> It is comparing apple and oranges. It is not about a "consistent 
> approach to how Java code interacts with native code" the approaches 
> of FFM and JNI are very different in fact.
>
> And that warning in its current form does definitely not make anything 
> safer.
>
>
> On 04.10.2023 21:29, Mark Reinhold wrote:
>> The following JEP is proposed to target JDK 22:
>>
>>    454: Foreign Function & Memory API
>>         https://openjdk.org/jeps/454
>>
>>    Summary: Introduce an API by which Java programs can interoperate with
>>    code and data outside of the Java runtime.  By efficiently invoking
>>    foreign functions (i.e., code outside the JVM), and by safely accessing
>>    foreign memory (i.e., memory not managed by the JVM), the API enables
>>    Java programs to call native libraries and process native data without
>>    the brittleness and danger of JNI.
>>
>> Feedback on this proposal from JDK Project Committers and Reviewers [1]
>> is more than welcome, as are reasoned objections.  If no such objections
>> are raised by 23:59 UTC on Wednesday, 11 October, or if they’re raised
>> and then satisfactorily answered, then per the JEP 2.0 process proposal
>> [2] I’ll target this JEP to JDK 22.
>>
>> - Mark
>>
>>
>> [1]https://openjdk.org/census#jdk
>> [2]https://cr.openjdk.java.net/~mr/jep/jep-2.0-02.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jdk-dev/attachments/20231009/b88b35b7/attachment-0001.htm>