New candidate JEP: 454: Foreign Function & Memory API

[Rony G. Flatscher]

Hi Ron,

first thank you very much for your interesting answer!

There seem to be a few problems that make the communication delude at times therefore trying to 
pointing them out upfront:

  * one is the term JRE which is supposed to identify the use case of a Java/JDK runtime installed
    in a system that gets used by any Java and non-Java application in each process; what you seem
    to be focusing on is only the "CRE" (customized - "embedded" - runtime environment) use case
    where an application developer determines/controls the configuration of the CRE s/he uses for
    her/his Java application,

  * another is the error message and who gets addressed by it introduced with FFM: the target use is
    the "application user" i.e. the application developer. From the discussions it became clear that
    the (end) user was never meant, but gets exposed to this message at JVM loading time not only if
    FFM is in use but also - all of a sudden and out of the blue after decades of deployments! - if
    JNI gets employed. As various posters have pointed out the proposed warning these (end) users
    get will be frightening (it will appear out of the blue for them once the JRE gets updated to
    the version that issues that warning) and they will most likely be afraid and not know what to
    do to remedy this unexpected, all of the sudden "dangerous" situation,

  * another is about the control over which runtime environment gets used, the JRE or the CRE, and
    who loads up the JVM: it seems that the expectation is that the "application author" is in full
    control over the JVM loading and that the only relevant use case is the CRE one, not the JRE
    one; as has been pointed out one of the biggest values of Java/JDK is its "compile once, run
    anywhere" supplying an incredible wealth of functionality in its JRE to whomever is able to
    exploit it: non-Java programs, non-Java tools got JNI by any of the Java JDKs in the past
    decades. The JRE deployment scenarios are manyfold, e.g. commands, utilities may be employed in
    a pipe using whatever runtime environment is present of which they may not know who started it
    in the first place. As such it is not possible to make sure that the warning does not get shown
    to the (end) user when there are different JRE's in use (even different JRE versions available
    on the same machine),

  * another one is "what is easy for whom" where the knowledge of this group seems to be the
    reference, but the knowledge of the Java application users/developers out in the field cannot be
    on par if they did not follow the project's discussions closely; add to that users who have no
    deep Java (language) knowledge at all and really constitute end users, people who work in
    departments on a computer and use their acquired programming skills to ease their business
    processes by taking advantage of them (like the PDFBox example).

On 14.09.2023 17:20, Ron Pressler wrote:
>> On 14 Sep 2023, at 13:21, Rony G. Flatscher<Rony.Flatscher at wu.ac.at>  wrote:
>>
>> As has been mentioned multiple times in the other e-mail thread on this particular planned warning it is really important to shelter these kind of users from that warning. There have been various constructive suggestions how one could go about it. Please do not ignore!
>> ---rony
>
> We do offer simple ways to shelter application users from warnings. Even when a Java runtime is shared, the application needs to create a JVM process (e.g. as you do on line 4569 of BSF4ooRexx.cc), at which point it can configure the JVM as it wishes, including to suppress the warning.

Actually the code already is able to configure the JVM. However, this can be done only, if 
BSF4ooRexx loads the JVM (and the environment variable BSF4Rexx_JavaStartupOptions is set). It 
cannot be done if the JVM got already loaded by someone else and the BSF4ooRexx bridge gets loaded 
into the process.

Also, the same programs may be run under different JRE versions (even on the same machine).

Add to that a scenario where ooRexx gets added as a Java scripting language (it implements 
javax.script) by adding it to an application by some (end) users who want to also be able to use 
ooRexx as a scripting language for their Java applications in addition to other scripting languages. 
(This probably applies to all SPI use cases where the configuration may be changed in the field by 
other users than the application authors.)

Of course, there are many more scenarios out in the field.

> The warning warns of a real (upcoming) incompatibility between the application and the Java runtime, and it is addressed to whoever it is that combines the application with the Java runtime because that’s who needs to handle such an incompatibility (the incompatibility is that the program is calling a restricted method without enabling native access, which will fail in a future release).

The problem is that there have been deployments that employ JNI in a stable and safe manner for a 
long time (some even for decades)!

Currently, there are no such jar's in the field that employ FFM (other than experimental), so there 
is an important deployment difference right here.

> As of JDK 9, the JDK provides the an application author with the ability to select a compatible runtime. That offers the application's user a superior experience to the one Java offered up until JDK 8. In that case, the warning — which is aimed at the person who picks the runtime — will be shown only to the application author (unless they choose not to suppress it).

Yes, that may be the case, but the point is that the implementations that use JNI are able to be 
executed in all versions of Java (at least starting from the level declared with JNI_version_x_y). 
There has been never really a need to pick a certain JRE for these use cases, so why invest time 
(develop, test, maintain) to support a nice feature if it does no good for the use case at hand?

> Also, just so that we’re clear on terminology, I assume that by “JRE” you mean a shared runtime, i.e. a Java runtime that’s intended to be shared among multiple applications (the JRE was retired five years ago and no longer exists).

An installation of OpenJDK makes Java available to all processes by default such that it constitutes 
a Java/JDK runtime environment (JRE) with which Java programs and native programs can interact with 
out of the box. This runtime property does not get changed just because file associations do not get 
set up for launchers etc. it continues to be a "Java/JDK runtime environment" (JRE).

Despite your claim that JRE does no longer exist, it exists and everyone can enjoy it by exploiting 
it to this day (since more than twenty years) in any of the processes.

> Now, not only did we not ignore the issue you raise, but we’d spent years considering and designing a solution before delivering it some years ago. Of course, the solution cannot solve the problem if you choose not to use it.
Well one needs to be made aware of it in the field such that those who create incompatibilities are 
responsible for communicating and for mitigating the effects as much as possible.
> If you care about an issue, consider the features and recommendations that Java has offered for the past years to address that very issue. It’s possible you have considered jlink and found a problem — if so, report it — but the fact that you refer to the JRE, a component that was retired some years ago, suggests that perhaps you have not fully considered its superior replacement (well, superior in most things if not all).

Terms like "superior" carry a grading but you leave out the concrete context. What is superiour of 
using JRE 21 over a supported JRE 17 over a supported JRE 8 for an application that was written for 
JRE 8? And if using JRE 21 why would JRE 22 all of a sudden bark up the wrong tree by issuing out of 
the blue a serious warning to (end) users who will be scared away from Java/JDK as a result!? For 
deployments and applications that have been working flawlessly for them for years if not decades 
already??


> I sense a fundamental misunderstanding that I think is at the core of the matter:
>
> An embedded runtime could do whatever a shared runtime can do and what the JRE could do back when it existed.
Of course.
> In fact, the application gets to choose what its runtime can do.
Why would one want to choose, if the installed JRE already has everything the application needs? Why 
investing time in something that is nice to have available but not needed for a certain use case?
> In particular, if the application so chooses, an embedded runtime could run arbitrary user-provided Java code.
That is also possible for any JRE?
> It is *not* necessarily tied to a specific set of classes. Indeed, even the runtime embedded in the JDK is generated in the same way that any other application or SDK can generate their own embedded Java runtimes, i.e. with jlink.
Again, why would one want to do that, if a JRE is available already that allows for everything? 
Which also has the benefit to save space and to have central (JRE) security updates to take 
immediately advantage of in the case you have many different Java applications which all use the 
same JRE? Etc.
> Each and every one of the use cases you listed will offer a superior experience if it went down this recommended path.
"Superior experience"? This another (positive) emotional term without context. For whom superior, 
and why superior (with respect to which alternatives)? Just stating an emotion is not (yet) 
sufficient to understand what is really meant.
> An application like LibreOffice can — and should! — use an embedded Java runtime to run users’ Java code.

Why should LibreOffice add another 150+, 200+, ... MB to its installation package if a JRE is 
already present??? There would be no benefit coming with this additional baggage. In effect 
OpenOffice/LibreOffice allow you to pick the JRE from those that are available on your computer.

The technical term BTW in the respective settings dialog label that lists the available JRE's is 
(cited verbatimly):

    "Java runtime environments (JRE) already installed:"

> Same goes for a Java2D Python library and for BSF4ooRexx. Their users' experience will be better.

Why should there be a proper CRE created, if any JRE that happens to be installed is already more 
than sufficient???

Why cripple down the JRE to a CRE that contains only the Java2D relevant modules? "Crippling" may 
seem harsh, but in effect that is what you ask for: rather than being able to exploit everything 
that the plain vanilla JRE offers, users get restricted to use only what an arbitrary CRE contains.

"Their user's experience will be better." - clearly not! Their user experience would be one of 
getting restricted, it would be worse!

If security issues make an update to the runtime environment necessary, the user experience is 
better if it is only needed to update the currently installed JRE which resolves the security issues 
for all the (Java, native) clients of that JRE.

> However, if application authors choose not to avail themselves of the ability to pick a runtime and insist on foisting on their users the selection of an appropriate Java runtime — a task that’s been made more onerous since the discontinuation of the JRE five years ago — then yes, it will suffer from the same shortcoming of the JRE age (and worse, because today’s shared runtimes do not offer some important client JRE features).

Again, it seems that you do not understand the problem at heart: there are cases where Java 
application authors want/need a customized runtime environment (CRE), but these cases are not the 
norm in the field. Rather, the standard deployment being to share and to be able to freely exploit 
all what an installed Java/JDK offers via its JRE and doing so in a platform independent matter, and 
in any combination that the (end) users see fit to best help solve their problems. It is the JRE 
deployment case that will cause the problem with that warning aimed at application authors getting 
shown erroneously to the (end) users instead.

In my corner I have to do among other things with ongoing business administration students who get 
empowered with programming skills and who have become able to exploit all that a JRE (sic!) offers 
in an easy manner. Among other things they learn to create/edit word processor documents, 
spreadsheet documents, presentation documents and the like using the UNO (universal network object) 
Java APIs of OpenOffice/LibreOffice (which BTW allows them to also work with MS Office documents in 
a platform independent manner).

It is one of many use cases for which this warning is not meant for.

(BTW there are also quite a few large sites that employ the ooRexx-Java bridge for which the 
occurrence of that warning will cause them to shy away from Java/JDK if Java/JDK itself starts to 
seriously warn about using something that had no problems for decades. It will help those fractions 
that are opposed to Java/JDK and prefer other environments.)

You cannot solve the effect that this new warning introduces by merely stating that this is the 
fault of all those who do not adhere to the latest, greatest rules, that get imposed on use cases 
and infrastructures that have been developed for more than two decades and therefore exist in the field!

The adverse effect if exposing (end) users to a warning meant to be addressed to an application 
user/author/developer is that it very likely will destroy Java's reputation quickly and effectively, 
something that seems to be hard to image for some without experiences in the field. There were quite 
a few authors that tried to communicate this particular problem in JRE-deployment scenarios on the 
other mail thread.

> Those users, who the application author entrusts with selecting a Java runtime for the application, *must* be warned about an incompatibility between the application code and the runtime they’ve chosen.

Why *must*?

What has changed all of a sudden when applying JNI safely for many years (even decades) that now the 
(end) users *must* be warned?

There is no technical incompatibility at all.

> Put simply, if you, an application owner, pick the runtime, then only you will be the one seeing the warning.

The point here is, which interestingly cannot be understood so far: there is no central "application 
owner" in the mentioned JRE deployment use cases for non-monolithic Java applications or programs. 
There are usages where the combination of native and Java code cannot be foreseen, let alone being 
able to identify a single "application owner". Think e.g. of commands, pipes in this context.

The term "application owner" only makes sense if you confine your thoughts to the use case in which 
a single, monolithic Java application gets created for which modules can be left out and added, such 
that it may make sense to create a custom ("embedded") runtime environment (CRE).

Why not have that particular FFM warning issued prominently to such an "application owner" when a 
CRE gets created (jlink), a Java project gets compiled (javac) or an executable jar (jar) created 
and then later, when you desire to force the "application author" to react upon your warning by 
changing the warning to an error for the "application author", i.e. when using jlink, javac and jar?

---

In addition why not think of a "module author" who can communicate the fact that FFM gets employed 
with an appropriate explicit statement in its module-info?

In addition if not a module but a plain jar, why not honor the usage to be explicitly communicated 
via the jar's MANIFEST.MF?

Also, why not come up with a utility that warns the application author that his application uses FFM 
without documenting this fact (and maybe other security warnings)?

> But if you choose not to do that and are asking me, your application user, to pick a runtime for your application, I need to be warned when I’ve picked an incompatible one.

One important property of Java has always been that any new version has tried to keep backward 
compatibility with all efforts. I expect that future versions do the same and only break backward 
compatibility if there is a very, very good reason for it. I fail to see such a very, very good 
reason here for the (end) users: if they have been able to safely and reliably exploit JNI one way 
or the other (some even for many decades!), I expect that future Java versions keep them running 
without a warning placed to the (end) users.

In the case of JNI it is obvious that it gets used by the fact that platform dependent binaries are 
needed. Therefore there is no real need for warning an application author in the context of JNI.

In the case of the new FFM this (using native code/memory) is not obvious anymore for an application 
author and therefore it may make sense to make sure that application authors are really aware of 
this fact.

> You’re complaining about how scary a red beach flag is when the actual danger is the water.

No, I am not warning about a red beach flag, I am warning *intensively* against someone crying 
"wolf!" for no good reason, frightening non-Java savvy people who are responsible for the good image 
and good afterwords, being the ambassadors in their companies for Java/JDK and its great openness 
and features. People who cannot assess the problem (short of the necessary background knowledge) and 
are not able to solve the problem on their own!

Once the message/warning gets issued by Java itself to (end) users that something dangerous is all 
of a sudden happening in native/Java programs that used to work correctly, securely and without 
problems for years. This not only insecures, but it destroys the trust in Java, their Java image of 
stableness, safety and easiness. They will be slapped in their face if Java itself warns from being 
used, when they have championed Java in the past.

Once an image damage via emotions (risky, dangerous, eavesdropping, virus, crashes, etc.) has been 
stated it can hardly be corrected and you will (inadvertently) cause more Java/JDK users (and their 
companies) to leave the Java/JDK world for good (and for no good reason)!

> You can choose to build a nice swimming pool for your users or you can choose to let them wade into the ocean, but letting them wade into the ocean without appropriate warning is just irresponsible.
By crying "wolf" you inhibit them using the road they have walked along for many decades, immediately.

> So either use an embedded runtime and offer your users a much improved experience, or not, and accept that they will need to be warned about issues that they — not you — will have to face as a result of your choice.

(That sentence could be translated to read: you either do what we say - no matter what it means for 
you and your users - or get lost.)

So you are saying that no one should use a JRE?
That everyone who wishes to use Java/JDK *must* create a CRE ("embedded runtime") all of a sudden 
(in order to become able to add that flag)?
And if not you will make sure that then the programs will cause warnings to (end) users and later 
not work for them anymore?

How come?

Please take JRE deployment cases as Java/JDK first citizens!

---

If you think of an "application author" to be one who is supposed to do something about an 
incompatibility that you introduce then please realize that there are no such "application authors" 
in these JRE deployment cases! We are talking about class libraries employing JNI that get used in 
various JRE deployment cases in which "application authors" in your sense do not exist.

Please re-read the use case examples presented by other authors in the other e-mail thread regarding 
this warning. This is a real problem!

> Having said all that, we’ll consider adding a line to the warning saying, “If you’re not the author of this application, please report this issue to the application vendor."

What is an (end) user supposed to do then?

What is a class library author using JNI supposed to do then? S/He can currently do nothing to 
alleviate this situation in JRE deployment cases.

---

The purpose of this warning it seems is to force an application author to acknowledge that s/he is 
aware of FFM being used for her/his application. Nothing less, nothing more.

This warning proposal came with FFM, and from the other discussions the intent is to make the 
application author/user/developer aware that s/he is using modules that employ FFM (which - unlike 
using JNI - may not be obvious to them).  To not be carried away it should be confined to the 
application author/user/developer in a way that saves the (end) users of ever seeing it.

---rony


-------- Forwarded Message --------
Subject: 	Re: New candidate JEP: 454: Foreign Function & Memory API
Date: 	Wed, 13 Sep 2023 18:33:07 +0200
From: 	Rony G. Flatscher <Rony.Flatscher at wu.ac.at>
To: 	jdk-dev at openjdk.org



On 11.09.2023 22:21, Mark Reinhold wrote:
> https://openjdk.org/jeps/454
>
>    Summary: Introduce an API by which Java programs can interoperate with
>    code and data outside of the Java runtime. By efficiently invoking
>    foreign functions (i.e., code outside the JVM), and by safely accessing
>    foreign memory (i.e., memory not managed by the JVM), the API enables
>    Java programs to call native libraries and process native data without
>    the brittleness and danger of JNI.

This JEP includes the "|--enable-native-access=M" and "||/.../ |the JAR-file manifest attribute 
|Enable-Native-Access: ALL-UNNAMED| can be used /in an executable JAR//.../"||

Can you explain how to inhibit the warning in JRE (Java/JDK runtime environment) deployment cases 
where neither a monolithic Java app nor an executable jar causes the JVM to get started up?

---

Ad "JRE":

  * A "JRE deployment case" gets established whenever OpenJDK gets installed in a system and thereby
    creating a Java/JDK runtime environment available for each process. Such a JRE deployment case
    is the standard deployment of OpenJDK and allows any Java application to be run. It has the
    benefit that one can use the same JRE for different Java applications (saving a lot of space)
    and in the case that a security update is needed for the currently installed JRE (the current
    system wide OpenJDK installation) it is sufficient to update the JRE and fix the critical
    situation for all the Java applications running under it at once.

  * A JRE deployment allows for non-Java applications like OpenOffice/LibreOffice to start the JVM
    and interact with it via JNI (the "other interaction direction" - from native to Java - for
    where JNI is the only solution).

  * A JRE deployment also allows for creating (little) programs, utilities and commands in non-Java
    languages that interact with the JRE and which may even allow for other non-Java programs to
    take advantage of the Java/JDK infrastructure in a platform independent manner which makes this
    deployment type very attractive in addition.
    Here an example in which there is also a short video demonstration at the end showing how a
    Python program can easily take advantage of Java2D without having itself any direct connection
    to Java at all (it also demonstrates how one could exploit that same utility even from Java
    programs using simple string commands to exploit Java2D instead of directly interacting with the
    Java2D Java classes): https://zenodo.org/record/8003114 (it is a five minute video demonstrating
    how simple string commands can be used to create Java2D graphics including some small animation
    examples in a platform independent manner, i.e. all the samples work unchanged on Windows, Linux
    and macOS).

There are many more deployment scenarios, of course, where it is important to realize that 
non-Java-savvy (end) users directly or indirectly take advantage of a JRE and who would not be able 
to understand why a serious warning would be shown all of a sudden by Java/JDK, what it really means 
and how to resolve it.

The JRE deployment is important for mixed operating system shops where using Java/JDK insulates the 
users from being locked-in into a certain operating system. As one of many possible examples the 
following link points to a Bachelor thesis of a business administration student who researched the 
Java tool PDFBox from the ASF (Apache Software Foundation) in order to become able to create, 
change, process/analyze PDF files, including signing PDF files and validation of signed PDF files or 
proper PDF/A; this work is a nice source of "PDF-how-to" and every Java programmer would immediately 
become able/jump start to take advantage of PDFBox in pure Java as well: 
<https://wi.wu.ac.at/rgf/diplomarbeiten/BakkStuff/2023/202303_Wang_ooRexx_and_PDFBox.pdf>. All the 
nutshell samples there run unchanged on Windows, Linux and macOS thanks to the JRE! (The student 
would be overwhelmed if all of a sudden a serious warning occurs about the usage of JNI and would 
hardly be able to understand why and how to solve that problem.)

---

As has been mentioned multiple times in the other e-mail thread on this particular planned warning 
it is really important to shelter these kind of users from that warning. There have been various 
constructive suggestions how one could go about it. Please do not ignore!

---rony

P.S.: In the discussion on the other e-mail thread it has become clear that the user who is supposed 
to be addressed with that particular warning is supposed to be the application author to make 
her/him aware that FFM gets employed in modules s/he uses which may be overseen otherwise. Therefore 
the warning should be issued in the toolset s/he has to use to build the application (probably 
jlink, jar, javac).

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/panama-dev/attachments/20230915/1b235300/attachment-0001.htm>