JEP proposed to target JDK 22: 454: Foreign Function & Memory API

[Ron Pressler]

Hi.

* For the past several years, the Java runtime has been issuing warnings of a mismatch between a program or its settings and the runtime. I believe that all of the last 13 feature releases of the JDK had at least one such warning (it’s possible JDK 16 didn’t).

* For the past several years these warnings were directed at whoever it is that application authors task with composing their program and its runtime. These are the right people as those are the ones that must deal with any mismatch between the application and the runtime; even if there were no warnings and the application simply ran incorrectly, these would still be the people facing the problem. It is the application’s author, not the JDK, who chooses who exactly these people are. As Maurizio said, whoever it is that the application author chooses to configure the application runtime *today*, those who deal with the same mismatch warning/errors *today*, should also be the ones to allow or disallow the use of native code.

* The client JRE, which was a runtime that had an elaborate protocol intended to alleviate the runtime/application incompatibilities (JNLP) that inevitably occur when such a distribution model is used, was discontinued years ago.  The client JRE, and the elaborate solutions it offered to the runtime/application compatibility problem were replaced, years ago, with simpler and better ones. Even the source code for the client JRE is gone from modern JDKs’ source repos.

* All the use cases you listed are addressed in the same manner they have been over the past several years, whether the jlinked runtime used is embedded in the application or shared among multiple applications, and whether the runtime is configured with a startup script, executable JAR, configuration @files, or the Invocation API. In *all* of these situations the application can configure the runtime to run without a warning shown to users (and without a future failure when the warnings turn into errors).

Everything you mention, including “shocking” warnings and who sees them, has been Java’s status quo for years. This particular JEP is proposed to target the fourteenth feature release under the new model and with the same kind of warning, and voicing your dislike of a strategy that’s been with us for years, over and over, is not productive. We’ve heard your concerns, we had the same ones in mind when we offered solutions to them back in 2017, and we’ll continue keeping them in mind as we continuously try to improve user experience. 

— Ron


> On 10 Oct 2023, at 11:44, Rony G. Flatscher <Rony.Flatscher at wu.ac.at> wrote:
> 
> Hi Maurizio,
> 
> On 09.10.2023 17:51, Maurizio Cimadamore wrote:
>> 
>> I believe you have made your position clear a number of time, both on panama-dev and on jdk-dev.
>> 
> Hmm, I am not sure, there seems to be still one particular communication problem (see below).
>> 
>> It is fair to say that, while we are not dismissing your concerns, our assessment of the overal impact of the new mechanism on the wider ecosystem is different than yours. You do not believe that the application packager is the right person to grant access to native code being used by the application.
>> 
> No, I am *not* saying that!
> 
> The application packagers can add that flag, of course and if it is part of the packaging procedure it is not a problem for them. However, this only holds, *if* there is an application package with a properly configured launcher defined for a Java program.
> 
> What has been attempted to communicate is something very different: there are plentyful of Java programs that have no application package defined for them and do not have a launcher that would set that flag for them.
> 
> For this group of people the warning will be quite shocking as it states that using the Java programs has become dangerous, so dangerous that it is even planned to inhibit the use of those Java programs sometimes in the future!
> 
>> This has been discussed at length [2] and it seems that past attempts to convince you of the contrary were all unsuccessful.
>> 
> Again this is *not* the case: the FFM JEP is fine. The sole exception and the reason for this thread is the planned warning being given to the *wrong* people! (This has nothing to do with "convictions".)
> 
> That warning is per se of limited utility (how can an acknowledgement for suppressing a warning make any Java application "safer"?). The "good intention" is to have the application packager be made aware of that native code gets used in the case she/he is not aware of it (this can never be wrong, can it?).
> 
> The problem however is that this warning will be shown to the wrong persons as well (this is where it becomes wrong!), like those who have been running Java programs successfully for years and in the case these programs use native code then after switching to Java 22 a serious warning pops up all of a sudden, telling those surprised users that using their Java program has become dangerous. The danger is regarded to be so high that in a future Java version Java will not allow to run that Java program anymore!
> 
> If those Java users happen to update to that latter Java version directly then the situation is even worse for them as now their Java program, they rely on, does not run anymore but gets killed with an error message at launch time. These Java users would not know why that happens and that there was a fix they could apply themselves.
> 
> From a technical point of view this might not be regarded to be a big problem, from a social point of view it is disastrous for the perception of Java and the Java ecosystem, hence this feedback and warning from issuing that warning to the wrong people.
> 
>> So I'd rather not rehash the same discussion here, and keep this discussion focussed on FFM (which is the subject of this thread).
>> 
>> The restricted method mechanism in FFM has been available for a long time - first time I see the mechanism being described is here:
>> 
>> https://openjdk.org/jeps/412
>> 
>> (from a couple of years ago).
>> 
> There are so many (interesting) projects and unless one has a specific interest in a particular one, none of the e-mail discussions of those e-mail lists would be known outside of the respective groups.
> 
> This thread has started out in jdk-dev after it was communicated that the JEP would cause a warning and later an error to be issued in the context of using native code via FFM and in that context all of a sudden for using JNI. Of course, no discussions from a couple of years ago in another list would be known.
> 
>> Note that in Java 17 there were no warnings issued for FFM, just errors. Following some feedback [1] which discussed many of the topics also more recently touched in the JNI discussion, we have decided to issue warnings for usage of restricted methods, as a transitionary measure. The reasons for doing that were:
>> 
>> * there was no way to enable native access for executable jars (we are adding that now)
>> 
> +1
>> 
>> * there was no way to enable native access for modules defined programmatically (we have added that in Java 20)
>> 
> +1
>> 
>> * there were (as for JNI) concerns involving the lack of a mechanism to propagate the native access permissions from a module to its dependencies
>> 
>> After which, no further feedback has been received. To be clear, we believe that the above issues (and few others, such as adquate javadoc support, and availability of static analyzers) are still very important, and we do not plan to turn the warnings into errors before all the above issues are addressed in a way that is satisfactory.
>> 
> +1
> 
> But please also take into account that all of this is potentially not known outside of the FFM group.
> 
>> You ask:
>> 
>>> Please pardon: what has JNI to do with FFM? 
>> 
>> I think there are a lot of things that JNI and FFM have in common, and there's also some things they do NOT have in common. For instance, JNI is much more unsafe, because, using its JNI functions, it can access any fields or methods bypassing the access checks that would normally be enforced by the Java runtime.
>> 
> Yes, I understand that FFM adheres to the access checks (and this is of course fine, appropriate). However, JNI predating these, being able to access anything from day one has also caused a number of design decisions that exploit that feature intentionally (and in a safe way). This is one of the major differences why JNI is not comparable to FFM.
> 
>> But both JNI and FFM compromise memory safety - e.g. they can create a situation where a client of a ByteBuffer, or a MemorySegment crashes because the region of memory behind the byte buffer/memory segment is no longer there. The way they do this is different (JNI does that using another JNI function, while FFM does that using restricted methods, such as MemorySegment::reinterpret). Also, both JNI and FFM can trigger library loading which can itself (via library loading hook) cause yet more native code to be executed.
>> 
> Yes, it is always possible - even without native access capabilities - to create situations where programs behave destructively. In the case of JNI one would look out from the beginning to not get into those crashes - as probably experienced the hard way when learning C, C++ - as otherwise the JNI library would be rendered useless.
> 
> FFM is different as it becomes possible that non-C/++, pure Java programmers may inadvertently cause crashes. But even so, such Java programmers would not have read the FFM documentation as it prominently warns about this, hence using FFM and not really knowing what they do. This scenario is not one that one would expect from professional Java programmers, would it?  And if a crash occurs it cannot go unnoticed hence the immediate pressure to go after the problem at which time all FFM related interactions that may cause a crash would be checked upon.
> 
>> In the current world, there is _no way_ for applications (e.g. pure Java ones) to guarantee that memory safety is not going to be compromised by any of their 3rd, 4th or 5th party dependencies. We would like to change that. The mechanism by which native access  is granted, while consistent with similar flags (e.g. `--illegal-access=permit`, or, more recently `-XX:+EnableDynamicAgentLoading`) might not be perfect, and we we are committed to make it better (see above). But we strongly believe in a world where native access requires an opt-in.
>> 
> Then why not do the same for Java itself explicitly, i.e. the application packager must also add e.g. java.base to proof that she/he is aware of native access there?
> 
> Native access via JNI has been there from day one and is needed not only for Java itself, but for bridging Java with non-Java applications, needless to say. JNI has been one of those strategic Java success factors that has made Java so attractive from the beginning.
> 
> ... cut ...
> 
> For the scenarios where my launchers get employed I will make sure in the next version that argument will be set for any new installation (making the use of Java 22+ cumbersome in this respect) to make sure that the Java users do not get that warning.
> 
> Unfortunately, there are scenarios where this is out of control because others do the launch (or older installations without that launch argument), causing Java 22+ to show that warning to non-application packagers (!), scaring them and probably scaring them away from Java over time as the (wrong) impression will be that all of a sudden using Java has become (too) dangerous.
> 
> ---rony
> 
>