JEP proposed to target JDK 22: 454: Foreign Function & Memory API

[Rony G. Flatscher]

Hi Maurizio,

On 09.10.2023 17:51, Maurizio Cimadamore wrote:
>
> I believe you have made your position clear a number of time, both on panama-dev and on jdk-dev.
>
Hmm, I am not sure, there seems to be still one particular communication problem (see below).
>
> It is fair to say that, while we are not dismissing your concerns, our assessment of the overal 
> impact of the new mechanism on the wider ecosystem is different than yours. You do not believe 
> that the application packager is the right person to grant access to native code being used by the 
> application.
>
No, I am *not* saying that!

The application packagers can add that flag, of course and if it is part of the packaging procedure 
it is not a problem for them. However, this only holds, *if* there is an application package with a 
properly configured launcher defined for a Java program.

What has been attempted to communicate is something very different: there are plentyful of Java 
programs that have no application package defined for them and do not have a launcher that would set 
that flag for them.

For this group of people the warning will be quite shocking as it states that using the Java 
programs has become dangerous, so dangerous that it is even planned to inhibit the use of those Java 
programs sometimes in the future!

> This has been discussed at length [2] and it seems that past attempts to convince you of the 
> contrary were all unsuccessful.
>
Again this is *not* the case: the FFM JEP is fine. The sole exception and the reason for this thread 
is the planned warning being given to the *wrong* people! (This has nothing to do with "convictions".)

That warning is per se of limited utility (how can an acknowledgement for suppressing a warning make 
any Java application "safer"?). The "good intention" is to have the application packager be made 
aware of that native code gets used in the case she/he is not aware of it (this can never be wrong, 
can it?).

The problem however is that this warning will be shown to the wrong persons as well (this is where 
it becomes wrong!), like those who have been running Java programs successfully for years and in the 
case these programs use native code then after switching to Java 22 a serious warning pops up all of 
a sudden, telling those surprised users that using their Java program has become dangerous. The 
danger is regarded to be so high that in a future Java version Java will not allow to run that Java 
program anymore!

If those Java users happen to update to that latter Java version directly then the situation is even 
worse for them as now their Java program, they rely on, does not run anymore but gets killed with an 
error message at launch time. These Java users would not know why that happens and that there was a 
fix they could apply themselves.

 From a technical point of view this might not be regarded to be a big problem, from a social point 
of view it is disastrous for the perception of Java and the Java ecosystem, hence this feedback and 
warning from issuing that warning to the wrong people.

> So I'd rather not rehash the same discussion here, and keep this discussion focussed on FFM (which 
> is the subject of this thread).
>
> The restricted method mechanism in FFM has been available for a long time - first time I see the 
> mechanism being described is here:
>
> https://openjdk.org/jeps/412
>
> (from a couple of years ago).
>
There are so many (interesting) projects and unless one has a specific interest in a particular one, 
none of the e-mail discussions of those e-mail lists would be known outside of the respective groups.

This thread has started out in jdk-dev after it was communicated that the JEP would cause a warning 
and later an error to be issued in the context of using native code via FFM and in that context all 
of a sudden for using JNI. Of course, no discussions from a couple of years ago in another list 
would be known.

> Note that in Java 17 there were no warnings issued for FFM, just errors. Following some feedback 
> [1] which discussed many of the topics also more recently touched in the JNI discussion, we have 
> decided to issue warnings for usage of restricted methods, as a transitionary measure. The reasons 
> for doing that were:
>
> * there was no way to enable native access for executable jars (we are adding that now)
>
+1
>
> * there was no way to enable native access for modules defined programmatically (we have added 
> that in Java 20)
>
+1
>
> * there were (as for JNI) concerns involving the lack of a mechanism to propagate the native 
> access permissions from a module to its dependencies
>
> After which, no further feedback has been received. To be clear, we believe that the above issues 
> (and few others, such as adquate javadoc support, and availability of static analyzers) are still 
> very important, and we do not plan to turn the warnings into errors before all the above issues 
> are addressed in a way that is satisfactory.
>
+1

But please also take into account that all of this is potentially not known outside of the FFM group.

> You ask:
>
>> Please pardon: what has JNI to do with FFM? 
>
> I think there are a lot of things that JNI and FFM have in common, and there's also some things 
> they do NOT have in common. For instance, JNI is much more unsafe, because, using its JNI 
> functions, it can access any fields or methods bypassing the access checks that would normally be 
> enforced by the Java runtime.
>
Yes, I understand that FFM adheres to the access checks (and this is of course fine, appropriate). 
However, JNI predating these, being able to access anything from day one has also caused a number of 
design decisions that exploit that feature intentionally (and in a safe way). This is one of the 
major differences why JNI is not comparable to FFM.

> But both JNI and FFM compromise memory safety - e.g. they can create a situation where a client of 
> a ByteBuffer, or a MemorySegment crashes because the region of memory behind the byte 
> buffer/memory segment is no longer there. The way they do this is different (JNI does that using 
> another JNI function, while FFM does that using restricted methods, such as 
> MemorySegment::reinterpret). Also, both JNI and FFM can trigger library loading which can itself 
> (via library loading hook) cause yet more native code to be executed.
>
Yes, it is always possible - even without native access capabilities - to create situations where 
programs behave destructively. In the case of JNI one would look out from the beginning to not get 
into those crashes - as probably experienced the hard way when learning C, C++ - as otherwise the 
JNI library would be rendered useless.

FFM is different as it becomes possible that non-C/++, pure Java programmers may inadvertently cause 
crashes. But even so, such Java programmers would not have read the FFM documentation as it 
prominently warns about this, hence using FFM and not really knowing what they do. This scenario is 
not one that one would expect from professional Java programmers, would it?  And if a crash occurs 
it cannot go unnoticed hence the immediate pressure to go after the problem at which time all FFM 
related interactions that may cause a crash would be checked upon.

> In the current world, there is _no way_ for applications (e.g. pure Java ones) to guarantee that 
> memory safety is not going to be compromised by any of their 3rd, 4th or 5th party dependencies. 
> We would like to change that. The mechanism by which native access  is granted, while consistent 
> with similar flags (e.g. `--illegal-access=permit`, or, more recently 
> `-XX:+EnableDynamicAgentLoading`) might not be perfect, and we we are committed to make it better 
> (see above). But we strongly believe in a world where native access requires an opt-in.
>
Then why not do the same for Java itself explicitly, i.e. the application packager must also add 
e.g. java.base to proof that she/he is aware of native access there?

Native access via JNI has been there from day one and is needed not only for Java itself, but for 
bridging Java with non-Java applications, needless to say. JNI has been one of those strategic Java 
success factors that has made Java so attractive from the beginning.

... cut ...

For the scenarios where my launchers get employed I will make sure in the next version that argument 
will be set for any new installation (making the use of Java 22+ cumbersome in this respect) to make 
sure that the Java users do not get that warning.

Unfortunately, there are scenarios where this is out of control because others do the launch (or 
older installations without that launch argument), causing Java 22+ to show that warning to 
non-application packagers (!), scaring them and probably scaring them away from Java over time as 
the (wrong) impression will be that all of a sudden using Java has become (too) dangerous.

---rony