From ron.pressler at oracle.com Thu Mar 16 18:48:13 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Thu, 16 Mar 2023 18:48:13 +0000 Subject: Disallowing the dynamic loading of agents by default Message-ID: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> Hi. In JDK 21 we intend to disallow the dynamic loading of agents by default. This will affect tools that use the Attach API to load an agent into a JVM some time after the JVM has started [1]. There is no change to any of the mechanisms that load an agent at JVM startup (-javaagent/-agentlib on the command line or the Launcher-Agent-Class attribute in the main JAR's manifest). This change in default behavior was proposed in 2017 as part of JEP 261 [2][3]. At that time the consensus was to switch to this default not in JDK 9 but in a later release to give tool maintainers sufficient time to inform their users. To allow the dynamic loading of agents, users will need to specify -XX:+EnableDynamicAgentLoading on the command line. I'll post a draft JEP for review shortly. -- Ron [1]: https://docs.oracle.com/en/java/javase/19/docs/api/jdk.attach/com/sun/tools/attach/package-summary.html [2]: https://openjdk.org/jeps/261 [3]: https://mail.openjdk.org/pipermail/jigsaw-dev/2017-April/012040.html From ron.pressler at oracle.com Fri Mar 17 13:42:03 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Fri, 17 Mar 2023 13:42:03 +0000 Subject: [External] : Re: Disallowing the dynamic loading of agents by default In-Reply-To: References: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> Message-ID: On 17 Mar 2023, at 13:33, Thomas St?fe > wrote: Hi Ron, Will this affect attaching via jcmd? The Attach mechanism will not be disabled by default, just the ability to load agents via the Attach mechanism. So the only jcmd command that will be affected is JVMTI.agent_load. To see the effect of the change today, launch java with -XX:-EnableDynamicAgentLoading, which is to become the new default. ? Ron -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron.pressler at oracle.com Fri Mar 17 14:29:56 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Fri, 17 Mar 2023 14:29:56 +0000 Subject: [External] : Re: Disallowing the dynamic loading of agents by default In-Reply-To: References: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> Message-ID: <8639867E-0D0A-402D-8A3B-41BB12A8F0BF@oracle.com> > On 17 Mar 2023, at 14:11, Thomas St?fe wrote: > > Thank you for the clarification. > > Oddly enough, -XX:-EnableDynamicAgentLoading seems to be broken. Tried head (fastdebug, release) and JDK17, even with this switch my sample library loads just fine: > > ``` > thomas at starfish$ ./images/jdk/bin/java -XX:-EnableDynamicAgentLoading -XX:+PrintFlagsFinal -cp $REPROS_JAR de.stuefe.repros.Simple > [Global flags] > ... > bool EnableDynamicAgentLoading = false {product} {command line} > ... > > OnAttach! Loading JVMTI sample agent > ``` > > Investigation shows that there seems to be a bug in attachListener.cpp where we compare AttachOperation::name for "load", but it contains "jcmd": > > ``` > Thread 22 "Attach Listener" hit Breakpoint 1, attach_listener_thread_entry (thread=0x7fff94000fd0, __the_thread__=0x7fff94000fd0) at /shared/projects/openjdk/jdk-jdk/source/src/hotspot/share/services/attachListener.cpp:404 > 404 } else if (!EnableDynamicAgentLoading && strcmp(op->name(), "load") == 0) { > (gdb) p op > $1 = (AttachOperation *) 0x7fff7401b640 > (gdb) p *op > $2 = {> = {}, _vptr.AttachOperation = 0x7ffff7b61210 , _name = "jcmd\000", '\361' , , _arg = { > "JVMTI.agent_load /shared/projects/jvmti-sample/sample.so\000", '\361' ..., "\000", '\361' ..., "\000", '\361' ...}} > (gdb) p op->name() > $3 = 0x7fff7401b648 "jcmd" > ``` > > This was on Linux x64. > > So if people have been using -XX:-EnableDynamicAgentLoading to check their code, this may not have worked as intended. > > Cheers, Thomas There may be a missing check in JVMTIAgentLoadDCmd::execute in diagnosticCommand.cpp. Thank you for reporting this! ? Ron From Alan.Bateman at oracle.com Sat Mar 18 07:21:44 2023 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 18 Mar 2023 07:21:44 +0000 Subject: [External] : Re: Disallowing the dynamic loading of agents by default In-Reply-To: References: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> Message-ID: On 17/03/2023 14:11, Thomas St?fe wrote: > : > > Investigation shows that there seems to be a bug in attachListener.cpp > where we compare AttachOperation::name for "load", but it contains > "jcmd": When using the Attach API, the VirtualMachine.loadAgentXXX methods map to a "load" command. The Attach API, jstack, jmap ... pre-date the jcmd tool and have their own set of commands known to both the tool/client side and the VM side.? The "jcmd" command, used by the jcmd tool, works a bit like a HTTP upgrade. So different path, and you are right, jcmd JVMTI.agent_load was missed when EnableDynamicAgentLoading was added in JDK 9. I've created JDK-8304438 to track it. -Alan From Alan.Bateman at oracle.com Sun Mar 19 09:27:02 2023 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sun, 19 Mar 2023 09:27:02 +0000 Subject: Disallowing the dynamic loading of agents by default In-Reply-To: <51bd4634-eae5-3471-525e-3af71229c53c@oss.nttdata.com> References: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> <51bd4634-eae5-3471-525e-3af71229c53c@oss.nttdata.com> Message-ID: On 19/03/2023 02:51, Yasumasa Suenaga wrote: > : > > Can we change flag type of EnableDynamicAgentLoading to `manageable` > from `product`? If so, we can use JVMTI agent without rebooting system > when we encountered some troubles in production system. If manageable then it could be enabled at run-time with HotSpotDiagnosticMXBean.setVMOption (or jcmd VM.set_flag), so I think wouldn't change anything. The main issue with JVMTI agents loaded into a running VM is that they can do anything. Even if their capabilities were reduced (and many debugging capabilities are only available in the onload phase) it can still use JNI and bypass access control. So I think a difficult security vs. serviceability trade-off here. -Alan. From ron.pressler at oracle.com Mon Mar 20 10:10:49 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Mon, 20 Mar 2023 10:10:49 +0000 Subject: [External] : Re: Disallowing the dynamic loading of agents by default In-Reply-To: References: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> Message-ID: <0737FC9F-629E-41E4-BD56-8955B6142FC8@oracle.com> Hi. The majority of serviceability tools don?t require dynamically loading an agent, and the majority of applications never load an agent dynamically. True, there are some tools that will be affected, which is why the decision was to introduce the flag in JDK 9 and to announce this change, but change the default in a later version to give tools ample time to prepare their users. The rationale for this change then hasn?t changed, but will be reiterated in a JEP (we just wanted to announce this ahead of the JEP to give tool authors another reminder more than six months ahead of JDK 21). The only change between then and now is that even fewer use cases require dynamically loaded agents, and so the impact is even smaller. It is also true that, when starting an application you don?t know that you *will* need to load an agent, but in most situations you know that you might. E.g. processes that are too critical to bring down even for deep maintenance (although not many of these are written in modern version of Java anyone) or canary services that are under trial. The relatively few sophisticated users who know how to write ad-hoc agents can even opt to enable dynamic agent loading on all their servers; these users are better equipped to can weigh the risks and tradeoffs involved. Finally, some tools that require a dynamically loaded JVM TI agents, such as profilers that profile native code, are so tied to the VM's internals that the best place for them is in the JDK. If anything, the bigger problem is not that profilers are used too much in production, but too little, including less advanced ones that don?t require an agent. There is plenty of time to enhance the JDK?s built-in profiling capabilities ahead of demand. ? Ron On 20 Mar 2023, at 01:21, Andrei Pangin > wrote: Hi all, Serviceability has been one of the biggest Java strengths, but the proposed change is going to have a large negative impact on it. Disallowing dynamic agents by default means it will no longer be possible to attach a profiler to a running app in runtime. JFR cannot close this gap due to lack of capabilities modern Java profilers have (that's a separate topic though). When an issue happens with a live app, it's already too late to add a command line argument. Furthermore, it may not be even feasible to add an agent at startup in containerized applications. Starting profiler on demand from the host OS or from a sidecar is the only viable solution in these cases. Next, it's hard to predict beforehand what tools exactly might be useful for troubleshooting: e.g., one tool may be better for finding memory leaks, a different one for analyzing CPU performance. Adding all possible tools at startup does not seem a reasonable approach, especially when tools may conflict with each other. The most important aspect of dynamic agents is the possibility to make a special tool just in time for solving a particular problem. A typical example is to get a value of some field in a live app without dumping the entire 60 GB heap. Another common use case is hot patching for fixing trivial bugs or for adding debug logs dynamically. The prominent example is when the dynamic agent has proved irreplaceable aid in addressing the notorious log4j vulnerabilities CVE-2021-44228 and CVE-2021-45046. I would be grateful to know more about the reasons why we should give up all the above advantages of dynamic agents in their good and legitimate use cases. Thank you, Andrei ??, 16 ???. 2023??. ? 18:48, Ron Pressler >: Hi. In JDK 21 we intend to disallow the dynamic loading of agents by default. This will affect tools that use the Attach API to load an agent into a JVM some time after the JVM has started [1]. There is no change to any of the mechanisms that load an agent at JVM startup (-javaagent/-agentlib on the command line or the Launcher-Agent-Class attribute in the main JAR's manifest). This change in default behavior was proposed in 2017 as part of JEP 261 [2][3]. At that time the consensus was to switch to this default not in JDK 9 but in a later release to give tool maintainers sufficient time to inform their users. To allow the dynamic loading of agents, users will need to specify -XX:+EnableDynamicAgentLoading on the command line. I'll post a draft JEP for review shortly. -- Ron [1]: https://docs.oracle.com/en/java/javase/19/docs/api/jdk.attach/com/sun/tools/attach/package-summary.html [2]: https://openjdk.org/jeps/261 [3]: https://mail.openjdk.org/pipermail/jigsaw-dev/2017-April/012040.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From volker.simonis at gmail.com Mon Mar 20 12:16:59 2023 From: volker.simonis at gmail.com (Volker Simonis) Date: Mon, 20 Mar 2023 13:16:59 +0100 Subject: [External] : Re: Disallowing the dynamic loading of agents by default In-Reply-To: References: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> <0737FC9F-629E-41E4-BD56-8955B6142FC8@oracle.com> Message-ID: Hi Ron, I'm still missing convincing technical arguments for disallowing dynamic loading of agents. If the argument is security then I can only agree with previous answers in that an attacker needs local access with the same credentials like the attacked JVM. But once he has that, all bets are off anyway. If you plan for features/enhancements/optimizations that rely on not being able to dynamically load an agent (which I haven't heard off yet), I don't understand this change either. Because as long as a switch for enabling dynamic loading exists (and I haven' heard that you want to completely forbid it) the dynamic loading use case has to be supported anyway. Dynamic agent loading is one of the features which sets the OpenJDK apart from other languages, managed runtimes and even closely related platforms like for example GraalVM Native Image which don't support such a feature. The mere existence of tools which rely on it and which are in widespread productive use, demonstrates its usefulness. And it is always good to know you have this possibility in your toolbox for the worst case (e.g. our log4j-hotpatcher [1]). I also can't by your argument that "the relatively few sophisticated users who know how to write ad-hoc agents can even opt to enable dynamic agent loading on all their servers". It is *exactly* not the few sophisticated authors of dynamic agents who would need to enable them but instead the millions of ingenious end-users and administrators who bag for help once they run into trouble. The other way round makes much more sense to me - the few sophisticated users who know for sure that they will never need the help of dynamic agents are free to disable them at startup. Given the current arguments, for me the usefulness of dynamic agents outweigh their drawbacks by far. Of course every OpenJDK distributor is free to change the default settings of command line options at his sole discretion, but I don't currently see a compelling reason for doing this by default for the whole OpenJDK community. If you have future plans which rely on disabling/forbidding dynamic agents please let us know. Best regards, Volker [1] https://aws.amazon.com/blogs/opensource/hotpatch-for-apache-log4j/ On Mon, Mar 20, 2023 at 11:37?AM Jaroslav Bachorik wrote: > > Hi, > > On Mon, Mar 20, 2023 at 11:11?AM Ron Pressler wrote: >> >> Hi. >> >> The majority of serviceability tools don?t require dynamically loading an agent, and the majority of applications never load an agent dynamically. > > > The majority of the JDK built-in tools, I would say. What about eg. the JMC agent? > >> >> >> True, there are some tools that will be affected, which is why the decision was to introduce the flag in JDK 9 and to announce this change, but change the default in a later version to give tools ample time to prepare their users. The rationale for this change then hasn?t changed, but will be reiterated in a JEP (we just wanted to announce this ahead of the JEP to give tool authors another reminder more than six months ahead of JDK 21). The only change between then and now is that even fewer use cases require dynamically loaded agents, and so the impact is even smaller. > > > As a maintainer of one of such tools I can confidently say that this change will either kill the tool as the ease of use will be gone or the workaround (eg. using JAVA_TOOL_OPTIONS) will completely defeat the purpose of this change. Having to put a flag when starting the JVM to allow dynamic loading of agents sounds a bit nonsensical to me - it would be much easier to directly add the agent to the JVM startup and then implement a lightweight control protocol over socket/shared memory to enabled/disable the agent features dynamically. > >> >> >> It is also true that, when starting an application you don?t know that you *will* need to load an agent, but in most situations you know that you might. E.g. processes that are too critical to bring down even for deep maintenance (although not many of these are written in modern version of Java anyone) or canary services that are under trial. The relatively few sophisticated users who know how to write ad-hoc agents can even opt to enable dynamic agent loading on all their servers; these users are better equipped to can weigh the risks and tradeoffs involved. > > > Wouldn't having this enabled system-wide actually defeat the purpose of having this flag? Considering that the dynamic attach can be performed only on the same host under the same user as the target process there seems to be a very small chance of loading agents accidentally. In the end people would set up their systems to enabled dynamic agent loading via eg. JAVA_TOOL_OPTIONS and we will be in the same place as before, with the additional hurdle of setting everything up. > >> >> Finally, some tools that require a dynamically loaded JVM TI agents, such as profilers that profile native code, are so tied to the VM's internals that the best place for them is in the JDK. If anything, the bigger problem is not that profilers are used too much in production, but too little, including less advanced ones that don?t require an agent. There is plenty of time to enhance the JDK?s built-in profiling capabilities ahead of demand. > > > I think this is an overly optimistic view. It is *much more* difficult to enhance the JDK's built-in profiling capabilities than do the same in an external profiling agent. > > > Overall, I don't seem to understand the anticipated attack vectors this change is supposed to prevent. AFAIK, in order to do the dynamic agent load one needs to have full access to the target process. That means that there are more convenient and straightforward ways to do anything nefarious than loading a JVMTI agent. Am I missing some other usages where the JVMTI agent would actually give access to something which would be otherwise inaccessible considering that the attacher and attachee must be on the same host and under the same user? > > Cheers, > > -JB- > >> >> >> ? Ron >> >> On 20 Mar 2023, at 01:21, Andrei Pangin wrote: >> >> Hi all, >> >> Serviceability has been one of the biggest Java strengths, but the proposed change is going to have a large negative impact on it. >> >> Disallowing dynamic agents by default means it will no longer be possible to attach a profiler to a running app in runtime. JFR cannot close this gap due to lack of capabilities modern Java profilers have (that's a separate topic though). >> >> When an issue happens with a live app, it's already too late to add a command line argument. Furthermore, it may not be even feasible to add an agent at startup in containerized applications. Starting profiler on demand from the host OS or from a sidecar is the only viable solution in these cases. >> >> Next, it's hard to predict beforehand what tools exactly might be useful for troubleshooting: e.g., one tool may be better for finding memory leaks, a different one for analyzing CPU performance. Adding all possible tools at startup does not seem a reasonable approach, especially when tools may conflict with each other. >> >> The most important aspect of dynamic agents is the possibility to make a special tool just in time for solving a particular problem. A typical example is to get a value of some field in a live app without dumping the entire 60 GB heap. Another common use case is hot patching for fixing trivial bugs or for adding debug logs dynamically. The prominent example is when the dynamic agent has proved irreplaceable aid in addressing the notorious log4j vulnerabilities CVE-2021-44228 and CVE-2021-45046. >> >> I would be grateful to know more about the reasons why we should give up all the above advantages of dynamic agents in their good and legitimate use cases. >> >> Thank you, >> Andrei >> >> ??, 16 ???. 2023??. ? 18:48, Ron Pressler : >>> >>> Hi. >>> >>> In JDK 21 we intend to disallow the dynamic loading of agents by default. This >>> will affect tools that use the Attach API to load an agent into a JVM some time >>> after the JVM has started [1]. There is no change to any of the mechanisms that >>> load an agent at JVM startup (-javaagent/-agentlib on the command line or the >>> Launcher-Agent-Class attribute in the main JAR's manifest). >>> >>> This change in default behavior was proposed in 2017 as part of JEP 261 [2][3]. >>> At that time the consensus was to switch to this default not in JDK 9 but in a >>> later release to give tool maintainers sufficient time to inform their users. >>> To allow the dynamic loading of agents, users will need to specify >>> -XX:+EnableDynamicAgentLoading on the command line. >>> >>> I'll post a draft JEP for review shortly. >>> >>> -- Ron >>> >>> [1]: https://docs.oracle.com/en/java/javase/19/docs/api/jdk.attach/com/sun/tools/attach/package-summary.html >>> [2]: https://openjdk.org/jeps/261 >>> [3]: https://mail.openjdk.org/pipermail/jigsaw-dev/2017-April/012040.html >> >> From greggwon at cox.net Mon Mar 20 13:42:51 2023 From: greggwon at cox.net (Gregg Wonderly) Date: Mon, 20 Mar 2023 08:42:51 -0500 Subject: [External] : Re: Disallowing the dynamic loading of agents by default In-Reply-To: <0737FC9F-629E-41E4-BD56-8955B6142FC8@oracle.com> References: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> <0737FC9F-629E-41E4-BD56-8955B6142FC8@oracle.com> Message-ID: <08242F6D-170B-43BD-B7B6-7E4DFB7B4662@cox.net> After all, we do know that Oracle in fact knows about every single Java application, where it runs, where it?s deployed and what the future plans are for the same. Otherwise, how else could they know what changes need to be made in the platform, right? Gregg Wonderly > On Mar 20, 2023, at 5:10 AM, Ron Pressler wrote: > > Hi. > > The majority of serviceability tools don?t require dynamically loading an agent, and the majority of applications never load an agent dynamically. > > True, there are some tools that will be affected, which is why the decision was to introduce the flag in JDK 9 and to announce this change, but change the default in a later version to give tools ample time to prepare their users. The rationale for this change then hasn?t changed, but will be reiterated in a JEP (we just wanted to announce this ahead of the JEP to give tool authors another reminder more than six months ahead of JDK 21). The only change between then and now is that even fewer use cases require dynamically loaded agents, and so the impact is even smaller. > > It is also true that, when starting an application you don?t know that you *will* need to load an agent, but in most situations you know that you might. E.g. processes that are too critical to bring down even for deep maintenance (although not many of these are written in modern version of Java anyone) or canary services that are under trial. The relatively few sophisticated users who know how to write ad-hoc agents can even opt to enable dynamic agent loading on all their servers; these users are better equipped to can weigh the risks and tradeoffs involved. > > Finally, some tools that require a dynamically loaded JVM TI agents, such as profilers that profile native code, are so tied to the VM's internals that the best place for them is in the JDK. If anything, the bigger problem is not that profilers are used too much in production, but too little, including less advanced ones that don?t require an agent. There is plenty of time to enhance the JDK?s built-in profiling capabilities ahead of demand. > > ? Ron > >> On 20 Mar 2023, at 01:21, Andrei Pangin > wrote: >> >> Hi all, >> >> Serviceability has been one of the biggest Java strengths, but the proposed change is going to have a large negative impact on it. >> >> Disallowing dynamic agents by default means it will no longer be possible to attach a profiler to a running app in runtime. JFR cannot close this gap due to lack of capabilities modern Java profilers have (that's a separate topic though). >> >> When an issue happens with a live app, it's already too late to add a command line argument. Furthermore, it may not be even feasible to add an agent at startup in containerized applications. Starting profiler on demand from the host OS or from a sidecar is the only viable solution in these cases. >> >> Next, it's hard to predict beforehand what tools exactly might be useful for troubleshooting: e.g., one tool may be better for finding memory leaks, a different one for analyzing CPU performance. Adding all possible tools at startup does not seem a reasonable approach, especially when tools may conflict with each other. >> >> The most important aspect of dynamic agents is the possibility to make a special tool just in time for solving a particular problem. A typical example is to get a value of some field in a live app without dumping the entire 60 GB heap. Another common use case is hot patching for fixing trivial bugs or for adding debug logs dynamically. The prominent example is when the dynamic agent has proved irreplaceable aid in addressing the notorious log4j vulnerabilities CVE-2021-44228 and CVE-2021-45046. >> >> I would be grateful to know more about the reasons why we should give up all the above advantages of dynamic agents in their good and legitimate use cases. >> >> Thank you, >> Andrei >> >> ??, 16 ???. 2023??. ? 18:48, Ron Pressler >: >>> Hi. >>> >>> In JDK 21 we intend to disallow the dynamic loading of agents by default. This >>> will affect tools that use the Attach API to load an agent into a JVM some time >>> after the JVM has started [1]. There is no change to any of the mechanisms that >>> load an agent at JVM startup (-javaagent/-agentlib on the command line or the >>> Launcher-Agent-Class attribute in the main JAR's manifest). >>> >>> This change in default behavior was proposed in 2017 as part of JEP 261 [2][3]. >>> At that time the consensus was to switch to this default not in JDK 9 but in a >>> later release to give tool maintainers sufficient time to inform their users. >>> To allow the dynamic loading of agents, users will need to specify >>> -XX:+EnableDynamicAgentLoading on the command line. >>> >>> I'll post a draft JEP for review shortly. >>> >>> -- Ron >>> >>> [1]: https://docs.oracle.com/en/java/javase/19/docs/api/jdk.attach/com/sun/tools/attach/package-summary.html >>> [2]: https://openjdk.org/jeps/261 >>> [3]: https://mail.openjdk.org/pipermail/jigsaw-dev/2017-April/012040.html > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron.pressler at oracle.com Mon Mar 20 17:53:25 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Mon, 20 Mar 2023 17:53:25 +0000 Subject: [External] : Re: Disallowing the dynamic loading of agents by default In-Reply-To: References: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> <0737FC9F-629E-41E4-BD56-8955B6142FC8@oracle.com> Message-ID: Hi Kirk. While the JEP will reiterate the relevant considerations (and no one denies that dynamically loaded agents are not useful) that led to this change being announced some years ago, the purpose of my email was to announce it will finally take effect in JDK 21. All the discussions at time, over all the relevant areas, informed the design at the core of the platform and its evolution in the past five years, namely that the application must grant explicit consent to anything affecting integrity (i.e. guarantees you can trust). Unless something has changed dramatically since then, or some new information has come to light, reopening the discussions around past decisions that have shaped the platform?s current design are unlikely to yield different results. Many of those discussions are available on jigsaw-dev. It is because some people may be affected that JEP 261 postponed the changing of that default, so that everyone would have time to prepare and prepare their users. We?ve now given an extra six-month advance notice to give those who haven?t finished preparing their users the time to do so. This is an opportunity to remind everyone that other capabilities that similarly affect platform and application integrity ? such as JNI and Unsafe ? will also require the application?s consent on the command line ? not in JDK 21, but soon thereafter. ? Ron On 20 Mar 2023, at 17:02, Kirk Pepperdine > wrote: Hi Ron, On Mar 20, 2023, at 3:10 AM, Ron Pressler > wrote: Hi. The majority of serviceability tools don?t require dynamically loading an agent, and the majority of applications never load an agent dynamically. While I wouldn?t be surprised that the majority don?t load agents dynamically, I wouldn?t want to diminishes the importance of this capability for those that do make use of it. And I believe the number that do dynamically load might surprise you. But then, my data on this is likely highly biased. Do you have better data to support this view point? True, there are some tools that will be affected, which is why the decision was to introduce the flag in JDK 9 and to announce this change, but change the default in a later version to give tools ample time to prepare their users. The rationale for this change then hasn?t changed, but will be reiterated in a JEP (we just wanted to announce this ahead of the JEP to give tool authors another reminder more than six months ahead of JDK 21). The only change between then and now is that even fewer use cases require dynamically loaded agents, and so the impact is even smaller. Again, I?m not sure I see the data to support this. But then again, my view point remains highly biased. And I see an assumption that tools will be able to easily adapt to this change. I?m not sure that is entirely true. At least not in a way that in effect returns dynamic attach capabilities with a directly loaded proxy. It is also true that, when starting an application you don?t know that you *will* need to load an agent, but in most situations you know that you might. E.g. processes that are too critical to bring down even for deep maintenance (although not many of these are written in modern version of Java anyone) or canary services that are under trial. The relatively few sophisticated users who know how to write ad-hoc agents can even opt to enable dynamic agent loading on all their servers; these users are better equipped to can weigh the risks and tradeoffs involved. Again, I?m not sure I?d equate numbers to importance. As an analogy, there are very few people that know how to build cars and maybe more that know how to fix them but, there are certainly many many more than know how to use them. Finally, some tools that require a dynamically loaded JVM TI agents, such as profilers that profile native code, are so tied to the VM's internals that the best place for them is in the JDK. If anything, the bigger problem is not that profilers are used too much in production, but too little, including less advanced ones that don?t require an agent. There is plenty of time to enhance the JDK?s built-in profiling capabilities ahead of demand. At odds is that all profilers come with biases. I?ve always stressed that the first thing one needs to do with any profiler is discover it?s biases and then determine how that bias affects the results, how one should account for the bias, or even should another tool be used. To this point, JFR, a built in profiler, has significant biases. For example, allocation profiling, quite often, completely misses allocation hotspots for small objects that are not scalar replaced in all but trivial examples. But let's not pick on JFR because it is the tool of choice for many other things and other allocation profilers do have other biases such as altering JIT behavior that may cause EA to fail thus preventing otherwise eligible allocation hotspots from being scalar replaced. While one tool is blind, the other generates false positives. Knowing this, I can combine the JIT logs with profiler results to help offset the effects of the bias for the later profile.. however, I can?t do anything for the blind spot. Finally, if there is anything lesson to be learned from the migrations from 8 to 9 is that tooling is a huge anchor preventing people from upgrading. That JDK 8 is still in as widespread use as it is, is in no small part due to the extensive change in the tooling chain. In fact, a number of very useful tools simply didn?t survive leaving us with less desirable alternatives. The other historical data point that maybe of comparison is the introduction of generics into the language. While this slowed the adoption of JDK 5 (from 1.4.2), it had no where near the impact that the degradation of the observability/diagnostic tool chain had on the migration rates from 7 to 8 and then this huge impact of 9. In my opinion, we?ve learned enough from this migration to understand that we may need to re-evaluate decisions that were made prior to these learning. Kind regards, Kirk ? Ron On 20 Mar 2023, at 01:21, Andrei Pangin > wrote: Hi all, Serviceability has been one of the biggest Java strengths, but the proposed change is going to have a large negative impact on it. Disallowing dynamic agents by default means it will no longer be possible to attach a profiler to a running app in runtime. JFR cannot close this gap due to lack of capabilities modern Java profilers have (that's a separate topic though). When an issue happens with a live app, it's already too late to add a command line argument. Furthermore, it may not be even feasible to add an agent at startup in containerized applications. Starting profiler on demand from the host OS or from a sidecar is the only viable solution in these cases. Next, it's hard to predict beforehand what tools exactly might be useful for troubleshooting: e.g., one tool may be better for finding memory leaks, a different one for analyzing CPU performance. Adding all possible tools at startup does not seem a reasonable approach, especially when tools may conflict with each other. The most important aspect of dynamic agents is the possibility to make a special tool just in time for solving a particular problem. A typical example is to get a value of some field in a live app without dumping the entire 60 GB heap. Another common use case is hot patching for fixing trivial bugs or for adding debug logs dynamically. The prominent example is when the dynamic agent has proved irreplaceable aid in addressing the notorious log4j vulnerabilities CVE-2021-44228 and CVE-2021-45046. I would be grateful to know more about the reasons why we should give up all the above advantages of dynamic agents in their good and legitimate use cases. Thank you, Andrei ??, 16 ???. 2023??. ? 18:48, Ron Pressler >: Hi. In JDK 21 we intend to disallow the dynamic loading of agents by default. This will affect tools that use the Attach API to load an agent into a JVM some time after the JVM has started [1]. There is no change to any of the mechanisms that load an agent at JVM startup (-javaagent/-agentlib on the command line or the Launcher-Agent-Class attribute in the main JAR's manifest). This change in default behavior was proposed in 2017 as part of JEP 261 [2][3]. At that time the consensus was to switch to this default not in JDK 9 but in a later release to give tool maintainers sufficient time to inform their users. To allow the dynamic loading of agents, users will need to specify -XX:+EnableDynamicAgentLoading on the command line. I'll post a draft JEP for review shortly. -- Ron [1]: https://docs.oracle.com/en/java/javase/19/docs/api/jdk.attach/com/sun/tools/attach/package-summary.html [2]: https://openjdk.org/jeps/261 [3]: https://mail.openjdk.org/pipermail/jigsaw-dev/2017-April/012040.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron.pressler at oracle.com Mon Mar 20 17:58:33 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Mon, 20 Mar 2023 17:58:33 +0000 Subject: [External] : Re: Disallowing the dynamic loading of agents by default In-Reply-To: References: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> <0737FC9F-629E-41E4-BD56-8955B6142FC8@oracle.com> Message-ID: > On 20 Mar 2023, at 17:53, Ron Pressler wrote: > > While the JEP will reiterate the relevant considerations (and no one denies that dynamically loaded agents are not useful) Sorry, no one denies that dynamically loaded agents *are* useful :) From ron.pressler at oracle.com Tue Mar 21 12:40:03 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Tue, 21 Mar 2023 12:40:03 +0000 Subject: [External] : Re: Disallowing the dynamic loading of agents by default In-Reply-To: References: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> <0737FC9F-629E-41E4-BD56-8955B6142FC8@oracle.com> Message-ID: <4D0F2EC3-1EED-45AB-B2F7-86D4AFD8006A@oracle.com> Hi Volker. JEP 261 states: "The dynamic loading of JVM TI agents will be disabled by default in a future release. To prepare for that change we recommend that applications that allow dynamic agents start using the option -XX:+EnableDynamicAgentLoading to enable that loading explicitly." The purpose of my email was to announce that that change will be put into effect in JDK 21 and to give a final reminder to those who have not yet done so to follow the recommendation in JEP 261 to prepare for that change. The Project Jigsaw team made that decision at the time after considering the perspectives of performance experts, security experts, and tooling experts, but unless anyone has some *new* information to present, there is no point in replaying the same discussions. You can revisit at least some of the technical discussions on jigsaw-dev. I will summarise the salient aspects (all discussed at the time) in the forthcoming JEP but, briefly, dynamically loaded agents -- alongside JNI and Unsafe -- break integrity, the ability to guarantee certain invariants, which has various implications on performance, security, and code evolution. They don't always break integrity in the way a cursory contemplation would suggest, which is why you should study those discussions if you're interested in the subject. Since JEP 261, the JDK has been evolving under the assumption that integrity is preserved unless the application grants explicit consent for it to be broken. As far as security in particular is concerned, the point you made is *not* the relevant one to the implications that were considered by Project Jigsaw at the time. As a member of the Vulnerability Group you may want to discuss that particular aspect with the appropriate people. -- Ron > On 20 Mar 2023, at 12:16, Volker Simonis wrote: > > Hi Ron, > > I'm still missing convincing technical arguments for disallowing > dynamic loading of agents. > > If the argument is security then I can only agree with previous > answers in that an attacker needs local access with the same > credentials like the attacked JVM. But once he has that, all bets are > off anyway. > > If you plan for features/enhancements/optimizations that rely on not > being able to dynamically load an agent (which I haven't heard off > yet), I don't understand this change either. Because as long as a > switch for enabling dynamic loading exists (and I haven' heard that > you want to completely forbid it) the dynamic loading use case has to > be supported anyway. > > Dynamic agent loading is one of the features which sets the OpenJDK > apart from other languages, managed runtimes and even closely related > platforms like for example GraalVM Native Image which don't support > such a feature. The mere existence of tools which rely on it and > which are in widespread productive use, demonstrates its usefulness. > And it is always good to know you have this possibility in your > toolbox for the worst case (e.g. our log4j-hotpatcher [1]). > > I also can't by your argument that "the relatively few sophisticated > users who know how to write ad-hoc agents can even opt to enable > dynamic agent loading on all their servers". It is *exactly* not the > few sophisticated authors of dynamic agents who would need to enable > them but instead the millions of ingenious end-users and > administrators who bag for help once they run into trouble. The other > way round makes much more sense to me - the few sophisticated users > who know for sure that they will never need the help of dynamic agents > are free to disable them at startup. > > Given the current arguments, for me the usefulness of dynamic agents > outweigh their drawbacks by far. Of course every OpenJDK distributor > is free to change the default settings of command line options at his > sole discretion, but I don't currently see a compelling reason for > doing this by default for the whole OpenJDK community. If you have > future plans which rely on disabling/forbidding dynamic agents please > let us know. > > Best regards, > Volker > > [1] https://urldefense.com/v3/__https://aws.amazon.com/blogs/opensource/hotpatch-for-apache-log4j/__;!!ACWV5N9M2RV99hQ!I7QWWsAmQNvmFzektSGaq4lWBWuMxP5R8P6nSwxfugmyEpKOrd_Io64JBX9mD8PBHywYZ7gEbDumhe5MdiQz_QdvpQ$ > > On Mon, Mar 20, 2023 at 11:37?AM Jaroslav Bachorik wrote: >> >> Hi, >> >> On Mon, Mar 20, 2023 at 11:11?AM Ron Pressler wrote: >>> >>> Hi. >>> >>> The majority of serviceability tools don?t require dynamically loading an agent, and the majority of applications never load an agent dynamically. >> >> >> The majority of the JDK built-in tools, I would say. What about eg. the JMC agent? >> >>> >>> >>> True, there are some tools that will be affected, which is why the decision was to introduce the flag in JDK 9 and to announce this change, but change the default in a later version to give tools ample time to prepare their users. The rationale for this change then hasn?t changed, but will be reiterated in a JEP (we just wanted to announce this ahead of the JEP to give tool authors another reminder more than six months ahead of JDK 21). The only change between then and now is that even fewer use cases require dynamically loaded agents, and so the impact is even smaller. >> >> >> As a maintainer of one of such tools I can confidently say that this change will either kill the tool as the ease of use will be gone or the workaround (eg. using JAVA_TOOL_OPTIONS) will completely defeat the purpose of this change. Having to put a flag when starting the JVM to allow dynamic loading of agents sounds a bit nonsensical to me - it would be much easier to directly add the agent to the JVM startup and then implement a lightweight control protocol over socket/shared memory to enabled/disable the agent features dynamically. >> >>> >>> >>> It is also true that, when starting an application you don?t know that you *will* need to load an agent, but in most situations you know that you might. E.g. processes that are too critical to bring down even for deep maintenance (although not many of these are written in modern version of Java anyone) or canary services that are under trial. The relatively few sophisticated users who know how to write ad-hoc agents can even opt to enable dynamic agent loading on all their servers; these users are better equipped to can weigh the risks and tradeoffs involved. >> >> >> Wouldn't having this enabled system-wide actually defeat the purpose of having this flag? Considering that the dynamic attach can be performed only on the same host under the same user as the target process there seems to be a very small chance of loading agents accidentally. In the end people would set up their systems to enabled dynamic agent loading via eg. JAVA_TOOL_OPTIONS and we will be in the same place as before, with the additional hurdle of setting everything up. >> >>> >>> Finally, some tools that require a dynamically loaded JVM TI agents, such as profilers that profile native code, are so tied to the VM's internals that the best place for them is in the JDK. If anything, the bigger problem is not that profilers are used too much in production, but too little, including less advanced ones that don?t require an agent. There is plenty of time to enhance the JDK?s built-in profiling capabilities ahead of demand. >> >> >> I think this is an overly optimistic view. It is *much more* difficult to enhance the JDK's built-in profiling capabilities than do the same in an external profiling agent. >> >> >> Overall, I don't seem to understand the anticipated attack vectors this change is supposed to prevent. AFAIK, in order to do the dynamic agent load one needs to have full access to the target process. That means that there are more convenient and straightforward ways to do anything nefarious than loading a JVMTI agent. Am I missing some other usages where the JVMTI agent would actually give access to something which would be otherwise inaccessible considering that the attacher and attachee must be on the same host and under the same user? >> >> Cheers, >> >> -JB- >> >>> >>> >>> ? Ron >>> >>> On 20 Mar 2023, at 01:21, Andrei Pangin wrote: >>> >>> Hi all, >>> >>> Serviceability has been one of the biggest Java strengths, but the proposed change is going to have a large negative impact on it. >>> >>> Disallowing dynamic agents by default means it will no longer be possible to attach a profiler to a running app in runtime. JFR cannot close this gap due to lack of capabilities modern Java profilers have (that's a separate topic though). >>> >>> When an issue happens with a live app, it's already too late to add a command line argument. Furthermore, it may not be even feasible to add an agent at startup in containerized applications. Starting profiler on demand from the host OS or from a sidecar is the only viable solution in these cases. >>> >>> Next, it's hard to predict beforehand what tools exactly might be useful for troubleshooting: e.g., one tool may be better for finding memory leaks, a different one for analyzing CPU performance. Adding all possible tools at startup does not seem a reasonable approach, especially when tools may conflict with each other. >>> >>> The most important aspect of dynamic agents is the possibility to make a special tool just in time for solving a particular problem. A typical example is to get a value of some field in a live app without dumping the entire 60 GB heap. Another common use case is hot patching for fixing trivial bugs or for adding debug logs dynamically. The prominent example is when the dynamic agent has proved irreplaceable aid in addressing the notorious log4j vulnerabilities CVE-2021-44228 and CVE-2021-45046. >>> >>> I would be grateful to know more about the reasons why we should give up all the above advantages of dynamic agents in their good and legitimate use cases. >>> >>> Thank you, >>> Andrei >>> >>> ??, 16 ???. 2023??. ? 18:48, Ron Pressler : >>>> >>>> Hi. >>>> >>>> In JDK 21 we intend to disallow the dynamic loading of agents by default. This >>>> will affect tools that use the Attach API to load an agent into a JVM some time >>>> after the JVM has started [1]. There is no change to any of the mechanisms that >>>> load an agent at JVM startup (-javaagent/-agentlib on the command line or the >>>> Launcher-Agent-Class attribute in the main JAR's manifest). >>>> >>>> This change in default behavior was proposed in 2017 as part of JEP 261 [2][3]. >>>> At that time the consensus was to switch to this default not in JDK 9 but in a >>>> later release to give tool maintainers sufficient time to inform their users. >>>> To allow the dynamic loading of agents, users will need to specify >>>> -XX:+EnableDynamicAgentLoading on the command line. >>>> >>>> I'll post a draft JEP for review shortly. >>>> >>>> -- Ron >>>> >>>> [1]: https://docs.oracle.com/en/java/javase/19/docs/api/jdk.attach/com/sun/tools/attach/package-summary.html >>>> [2]: https://openjdk.org/jeps/261 >>>> [3]: https://mail.openjdk.org/pipermail/jigsaw-dev/2017-April/012040.html >>> >>> From adinn at redhat.com Fri Mar 24 17:21:15 2023 From: adinn at redhat.com (Andrew Dinn) Date: Fri, 24 Mar 2023 17:21:15 +0000 Subject: Disallowing the dynamic loading of agents by default In-Reply-To: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> References: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> Message-ID: Hi Ron, Thank you for providing a heads up on the proposed JEP. The Red Hat Java team have been discussing this proposal. We have reviewed the original discussion and also the surrounding debate which established requirements for adaptation of Jigsaw to incorporate the needs of agents. As an aside, I'll note that a thorough review was necessary /even/ in my case, despite the fact that I was an active party, because the discussion occurred, and corresponding decisions were made, quite some time ago. I mention this because it may explain the air of surprise and the desire to reiterate some of the original debate on the part of some respondents in this thread, who perhaps were not party, or only tangentially party, to the discussion. That also suggests that there may be a lot users who are not aware that the -XX:+EnableDynamicAgentLoading switch exists or do not really understand why it exists i.e. that there is a broad education issue at play here. We do have some concerns about the JEP, specifically about the timing of its delivery. These are probably best addressed via the normal review process. In particular that will ensure the discussion happens in a more suitable and more widely subscribed forum than the Jigsaw list. However, I will briefly mention our concerns in this reply. Before that let me start with a few disclaimers: - We acknowledge that there is little to be gained from re-iterating arguments made in the previous discussion (although that does not imply the JEP review would not benefit from new arguments, especially from those who were not involved in that discussion) - We recognize that the purpose of the -XX:+EnableDynamicAgentLoading switch is to offer a platform integrity guarantee and that this change of the default reflects a desire to prioritise integrity over the flexibility that agents provide - We recognize that the proposal is only proposing to flip a configuration default rather than detract from (or modify) available functionality - We recognize that changing this default will still allow (*most*) users to configure the behaviour they desire - We recognize that this advance notice has been given precisely to ensure that anyone wishing to deploy on jdk21 an app that relies on use of agents has time to plan appropriate configuration for their deployment - We recognize that this change of default is not being proposed for backport and hence that it will largely only affect the relatively small number of users who are currently developing for jdk21+ So, given that as a base for our comments where is the beef? - Our main concern is, predictably, timing. Clearly, this is a future, potential problem rather than a present problem - no one can be deploying on jdk21 yet and most developers who are currently preparing an app for deployment on jdk21+ will likely encounter the effect of this change before actual deployment and be in a position to remedy it. The concern is that advertising a change like this and getting users prepared to respond to it has always been difficult to achieve. In particular we expect a long tail of support problems from users who are trying to upgrade deployments from earlier releases to jdk21. So, while it is nice to have such early notice of the proposal we plan to review its likely impact on our users and how much time we need to prepare ourselves and our users to negotiate this change in behaviour. Any evidence we obtain to suggest a delay in targeting is appropriate will be brought to the JEP review. - A second, related concern is that flipping the default for this configuration in an LTS release as the first exposure to it for most people is more likely to derail deployment plans for users than if the default were flipped in a non-LTS release. If this change were deferred to jdk22 then that would give those planning deployment on (or upgrade to) jdk25 and also those planning to upgrade from jdk17 to jdk21 more time to discover and respond to the change. - A third concern, already pointed out by Volker, is that some users may run their Java apps via launcher apps or scripts that mask access to the Java command line. For such users the change of default may mean that they lose the option to deploy dynamic agents for important ancillary tasks such as observability. We are not clear how many of our users this affects but we will be looking into this and hope to bring feedback to the JEP review. Obviously, this problem can be remedied relatively easily by the supplier of the launcher enabling agent use or providing a suitable control switch. Our concern is not with how to solve this problem rather how the involvement of two parties, supplier and end user, might imply a need for the JEP to be targeted to a later release. regards, Andrew Dinn ----------- Red Hat Distinguished Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill From greggwon at cox.net Fri Mar 24 19:56:32 2023 From: greggwon at cox.net (Gregg Wonderly) Date: Fri, 24 Mar 2023 14:56:32 -0500 Subject: Disallowing the dynamic loading of agents by default In-Reply-To: References: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> Message-ID: <33D9DA9C-5539-40FC-8AAF-21926C6CFE9B@cox.net> Lot?s of people use Java in places where there is no ?release? cycle of Java version in control of the users. These are ?corporate users? in most cases and they have Java applications that they are using which will just ?stop working? when a new version of Java is installed. Over the years, I?ve watch any favoritism towards java on the desktop or as a general solution programing language wane, because it?s undependable as a platform. You never no when something will break as these ?stability? changes occur. People who use software systems are in large part not programers or language/platform experts. The in ability of Oracle and many others to understand how detrimental this behavior has been is just mind blowing. People like myself end up looking like whining babies because we come back every once in a while to see if there is something useful happening in Java development that might finally stabilize the platform on the desktop and other business environments and low and behold write-once-run-anywhere is found to still be unimplemented and basically non appreciated. It?s just a sad, sad thing to see happening. Sun first did this with Java 1.2. The Community beat up on Sun severely and everything quieted down for a while. Then we had the JDK 1.5 release where my much mentioned volatile reachability optimizations broke software all over the place. This is not happening in any other language I am aware of. The people at Sun who were causing all the problems seemed to have gone on to Oracle and there is just a core group of people who just do not understand how horrible Java looks these days because of how much basic functionality got completely broken when a new version of Java showed up on general purpose computing and working software just stopped working? Gregg Wonderly > On Mar 24, 2023, at 12:21 PM, Andrew Dinn wrote: > > Hi Ron, > > Thank you for providing a heads up on the proposed JEP. The Red Hat Java team have been discussing this proposal. We have reviewed the original discussion and also the surrounding debate which established requirements for adaptation of Jigsaw to incorporate the needs of agents. > > As an aside, I'll note that a thorough review was necessary /even/ in my case, despite the fact that I was an active party, because the discussion occurred, and corresponding decisions were made, quite some time ago. I mention this because it may explain the air of surprise and the desire to reiterate some of the original debate on the part of some respondents in this thread, who perhaps were not party, or only tangentially party, to the discussion. > > That also suggests that there may be a lot users who are not aware that the -XX:+EnableDynamicAgentLoading switch exists or do not really understand why it exists i.e. that there is a broad education issue at play here. > > We do have some concerns about the JEP, specifically about the timing of its delivery. These are probably best addressed via the normal review process. In particular that will ensure the discussion happens in a more suitable and more widely subscribed forum than the Jigsaw list. However, I will briefly mention our concerns in this reply. Before that let me start with a few disclaimers: > > - We acknowledge that there is little to be gained from re-iterating arguments made in the previous discussion (although that does not imply the JEP review would not benefit from new arguments, especially from those who were not involved in that discussion) > > - We recognize that the purpose of the -XX:+EnableDynamicAgentLoading switch is to offer a platform integrity guarantee and that this change of the default reflects a desire to prioritise integrity over the flexibility that agents provide > > - We recognize that the proposal is only proposing to flip a configuration default rather than detract from (or modify) available functionality > > - We recognize that changing this default will still allow (*most*) users to configure the behaviour they desire > > - We recognize that this advance notice has been given precisely to ensure that anyone wishing to deploy on jdk21 an app that relies on use of agents has time to plan appropriate configuration for their deployment > > - We recognize that this change of default is not being proposed for backport and hence that it will largely only affect the relatively small number of users who are currently developing for jdk21+ > > So, given that as a base for our comments where is the beef? > > - Our main concern is, predictably, timing. Clearly, this is a future, potential problem rather than a present problem - no one can be deploying on jdk21 yet and most developers who are currently preparing an app for deployment on jdk21+ will likely encounter the effect of this change before actual deployment and be in a position to remedy it. The concern is that advertising a change like this and getting users prepared to respond to it has always been difficult to achieve. In particular we expect a long tail of support problems from users who are trying to upgrade deployments from earlier releases to jdk21. > So, while it is nice to have such early notice of the proposal we plan to review its likely impact on our users and how much time we need to prepare ourselves and our users to negotiate this change in behaviour. Any evidence we obtain to suggest a delay in targeting is appropriate will be brought to the JEP review. > > - A second, related concern is that flipping the default for this configuration in an LTS release as the first exposure to it for most people is more likely to derail deployment plans for users than if the default were flipped in a non-LTS release. If this change were deferred to jdk22 then that would give those planning deployment on (or upgrade to) jdk25 and also those planning to upgrade from jdk17 to jdk21 more time to discover and respond to the change. > > - A third concern, already pointed out by Volker, is that some users may run their Java apps via launcher apps or scripts that mask access to the Java command line. For such users the change of default may mean that they lose the option to deploy dynamic agents for important ancillary tasks such as observability. We are not clear how many of our users this affects but we will be looking into this and hope to bring feedback to the JEP review. > Obviously, this problem can be remedied relatively easily by the supplier of the launcher enabling agent use or providing a suitable control switch. Our concern is not with how to solve this problem rather how the involvement of two parties, supplier and end user, might imply a need for the JEP to be targeted to a later release. > > regards, > > > Andrew Dinn > ----------- > Red Hat Distinguished Engineer > Red Hat UK Ltd > Registered in England and Wales under Company Registration No. 03798903 > Directors: Michael Cunningham, Michael ("Mike") O'Neill > From greggwon at cox.net Sat Mar 25 19:56:50 2023 From: greggwon at cox.net (Gregg G Wonderly) Date: Sat, 25 Mar 2023 14:56:50 -0500 Subject: Disallowing the dynamic loading of agents by default In-Reply-To: <8DB1B340-1DBD-5A46-BE6C-24364031F5B4@hxcore.ol> References: <8DB1B340-1DBD-5A46-BE6C-24364031F5B4@hxcore.ol> Message-ID: <39D27A60-7914-4109-A19E-96AB5608A4F5@cox.net> I understand you may have personal experiences with how you use Java. In my experience and others, Java has constantly had fundamental breakage in various details due to lack of understanding, on the platform development team(s) of what people actually do with Java. Sent from my iPhone > On Mar 25, 2023, at 5:29 AM, Bernd wrote: > > ? > Gregg i have to disagree, not only is Java one of the most stable platform out there but also the ?Enterprise Desktop? couldn?t care less about the default of this switch. > > Gruss > Bernd > -- > http://bernd.eckenfels.net > > Von: serviceability-dev im Auftrag von Gregg Wonderly > Gesendet: Freitag, M?rz 24, 2023 11:42 PM > An: Andrew Dinn > Cc: Ron Pressler ; jigsaw-dev at openjdk.org ; serviceability-dev at openjdk.org > Betreff: Re: Disallowing the dynamic loading of agents by default > > Lot?s of people use Java in places where there is no ?release? cycle of Java version in control of the users. These are ?corporate users? in most cases and they have Java applications that they are using which will just ?stop working? when a new version of Java is installed. > > Over the years, I?ve watch any favoritism towards java on the desktop or as a general solution programing language wane, because it?s undependable as a platform. You never no when something will break as these ?stability? changes occur. People who use software systems are in large part not programers or language/platform experts. The in ability of Oracle and many others to understand how detrimental this behavior has been is just mind blowing. > > People like myself end up looking like whining babies because we come back every once in a while to see if there is something useful happening in Java development that might finally stabilize the platform on the desktop and other business environments and low and behold write-once-run-anywhere is found to still be unimplemented and basically non appreciated. It?s just a sad, sad thing to see happening. > > Sun first did this with Java 1.2. The Community beat up on Sun severely and everything quieted down for a while. Then we had the JDK 1.5 release where my much mentioned volatile reachability optimizations broke software all over the place. > > This is not happening in any other language I am aware of. The people at Sun who were causing all the problems seemed to have gone on to Oracle and there is just a core group of people who just do not understand how horrible Java looks these days because of how much basic functionality got completely broken when a new version of Java showed up on general purpose computing and working software just stopped working? > > Gregg Wonderly > > > On Mar 24, 2023, at 12:21 PM, Andrew Dinn wrote: > > > > Hi Ron, > > > > Thank you for providing a heads up on the proposed JEP. The Red Hat Java team have been discussing this proposal. We have reviewed the original discussion and also the surrounding debate which established requirements for adaptation of Jigsaw to incorporate the needs of agents. > > > > As an aside, I'll note that a thorough review was necessary /even/ in my case, despite the fact that I was an active party, because the discussion occurred, and corresponding decisions were made, quite some time ago. I mention this because it may explain the air of surprise and the desire to reiterate some of the original debate on the part of some respondents in this thread, who perhaps were not party, or only tangentially party, to the discussion. > > > > That also suggests that there may be a lot users who are not aware that the -XX:+EnableDynamicAgentLoading switch exists or do not really understand why it exists i.e. that there is a broad education issue at play here. > > > > We do have some concerns about the JEP, specifically about the timing of its delivery. These are probably best addressed via the normal review process. In particular that will ensure the discussion happens in a more suitable and more widely subscribed forum than the Jigsaw list. However, I will briefly mention our concerns in this reply. Before that let me start with a few disclaimers: > > > > - We acknowledge that there is little to be gained from re-iterating arguments made in the previous discussion (although that does not imply the JEP review would not benefit from new arguments, especially from those who were not involved in that discussion) > > > > - We recognize that the purpose of the -XX:+EnableDynamicAgentLoading switch is to offer a platform integrity guarantee and that this change of the default reflects a desire to prioritise integrity over the flexibility that agents provide > > > > - We recognize that the proposal is only proposing to flip a configuration default rather than detract from (or modify) available functionality > > > > - We recognize that changing this default will still allow (*most*) users to configure the behaviour they desire > > > > - We recognize that this advance notice has been given precisely to ensure that anyone wishing to deploy on jdk21 an app that relies on use of agents has time to plan appropriate configuration for their deployment > > > > - We recognize that this change of default is not being proposed for backport and hence that it will largely only affect the relatively small number of users who are currently developing for jdk21+ > > > > So, given that as a base for our comments where is the beef? > > > > - Our main concern is, predictably, timing. Clearly, this is a future, potential problem rather than a present problem - no one can be deploying on jdk21 yet and most developers who are currently preparing an app for deployment on jdk21+ will likely encounter the effect of this change before actual deployment and be in a position to remedy it. The concern is that advertising a change like this and getting users prepared to respond to it has always been difficult to achieve. In particular we expect a long tail of support problems from users who are trying to upgrade deployments from earlier releases to jdk21. > > So, while it is nice to have such early notice of the proposal we plan to review its likely impact on our users and how much time we need to prepare ourselves and our users to negotiate this change in behaviour. Any evidence we obtain to suggest a delay in targeting is appropriate will be brought to the JEP review. > > > > - A second, related concern is that flipping the default for this configuration in an LTS release as the first exposure to it for most people is more likely to derail deployment plans for users than if the default were flipped in a non-LTS release. If this change were deferred to jdk22 then that would give those planning deployment on (or upgrade to) jdk25 and also those planning to upgrade from jdk17 to jdk21 more time to discover and respond to the change. > > > > - A third concern, already pointed out by Volker, is that some users may run their Java apps via launcher apps or scripts that mask access to the Java command line. For such users the change of default may mean that they lose the option to deploy dynamic agents for important ancillary tasks such as observability. We are not clear how many of our users this affects but we will be looking into this and hope to bring feedback to the JEP review. > > Obviously, this problem can be remedied relatively easily by the supplier of the launcher enabling agent use or providing a suitable control switch. Our concern is not with how to solve this problem rather how the involvement of two parties, supplier and end user, might imply a need for the JEP to be targeted to a later release. > > > > regards, > > > > > > Andrew Dinn > > ----------- > > Red Hat Distinguished Engineer > > Red Hat UK Ltd > > Registered in England and Wales under Company Registration No. 03798903 > > Directors: Michael Cunningham, Michael ("Mike") O'Neill > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aph-open at littlepinkcloud.com Mon Mar 27 09:41:42 2023 From: aph-open at littlepinkcloud.com (Andrew Haley) Date: Mon, 27 Mar 2023 10:41:42 +0100 Subject: Disallowing the dynamic loading of agents by default In-Reply-To: <33D9DA9C-5539-40FC-8AAF-21926C6CFE9B@cox.net> References: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> <33D9DA9C-5539-40FC-8AAF-21926C6CFE9B@cox.net> Message-ID: <9d087793-15d6-5a4f-df12-a254fbb0e829@littlepinkcloud.com> On 3/24/23 19:56, Gregg Wonderly wrote: > People like myself end up looking like whining babies because we come back every once in a while to see if there is something useful happening in Java development that might finally stabilize the platform Perfect stability will happen when we stop developing Java. In the meantime, we continue to support old releases via the Updates project, and people keep using Java 8. > This is not happening in any other language I am aware of. That I can believe. The rest of us, however, know that Java is one of the most stable platforms. If you'd like to be more aware of what happens to other popular languages, I suggest you Google "Python 2->3 transition". -- Andrew Haley (he/him) Java Platform Lead Engineer Red Hat UK Ltd. https://keybase.io/andrewhaley EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From ron.pressler at oracle.com Mon Mar 27 10:32:24 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Mon, 27 Mar 2023 10:32:24 +0000 Subject: [External] : Re: Disallowing the dynamic loading of agents by default In-Reply-To: References: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> Message-ID: <023F29ED-CAF3-4D32-B36C-8053DDCC580A@oracle.com> Hi Andrew! > On 24 Mar 2023, at 17:21, Andrew Dinn wrote: > > Hi Ron, > > Thank you for providing a heads up on the proposed JEP. The Red Hat Java team have been discussing this proposal. We have reviewed the original discussion and also the surrounding debate which established requirements for adaptation of Jigsaw to incorporate the needs of agents. > > As an aside, I'll note that a thorough review was necessary /even/ in my case, despite the fact that I was an active party, because the discussion occurred, and corresponding decisions were made, quite some time ago. I mention this because it may explain the air of surprise and the desire to reiterate some of the original debate on the part of some respondents in this thread, who perhaps were not party, or only tangentially party, to the discussion. > > That also suggests that there may be a lot users who are not aware that the -XX:+EnableDynamicAgentLoading switch exists or do not really understand why it exists i.e. that there is a broad education issue at play here. Understood. > > We do have some concerns about the JEP, specifically about the timing of its delivery. These are probably best addressed via the normal review process. In particular that will ensure the discussion happens in a more suitable and more widely subscribed forum than the Jigsaw list. However, I will briefly mention our concerns in this reply. Before that let me start with a few disclaimers: > > - We acknowledge that there is little to be gained from re-iterating arguments made in the previous discussion (although that does not imply the JEP review would not benefit from new arguments, especially from those who were not involved in that discussion) > > - We recognize that the purpose of the -XX:+EnableDynamicAgentLoading switch is to offer a platform integrity guarantee and that this change of the default reflects a desire to prioritise integrity over the flexibility that agents provide I would qualify that: we want to prioritise integrity *by default*. Integrity is only practical when it is the default, as adding more integrity rules after the fact is hard to the point of being impractical. Agents in themselves don?t impact that default, but dynamically loaded agents do. > > - We recognize that the proposal is only proposing to flip a configuration default rather than detract from (or modify) available functionality > > - We recognize that changing this default will still allow (*most*) users to configure the behaviour they desire > > - We recognize that this advance notice has been given precisely to ensure that anyone wishing to deploy on jdk21 an app that relies on use of agents has time to plan appropriate configuration for their deployment > > - We recognize that this change of default is not being proposed for backport and hence that it will largely only affect the relatively small number of users who are currently developing for jdk21+ > > So, given that as a base for our comments where is the beef? > > - Our main concern is, predictably, timing. Clearly, this is a future, potential problem rather than a present problem - no one can be deploying on jdk21 yet and most developers who are currently preparing an app for deployment on jdk21+ will likely encounter the effect of this change before actual deployment and be in a position to remedy it. The concern is that advertising a change like this and getting users prepared to respond to it has always been difficult to achieve. In particular we expect a long tail of support problems from users who are trying to upgrade deployments from earlier releases to jdk21. > So, while it is nice to have such early notice of the proposal we plan to review its likely impact on our users and how much time we need to prepare ourselves and our users to negotiate this change in behaviour. Any evidence we obtain to suggest a delay in targeting is appropriate will be brought to the JEP review. Very well, we can certainly discuss timing as part of the JEP discussion. The JEP itself is rather ambitious because I noticed that most of the strong encapsulation JEPs lacked a substantial motivation section (largely because they had been written before that became the norm) and so strong encapsulation has not been motivated in JEP form, and this is an opportunity to start rectifying that. This is important because some, even in this discussion, are under the impression that integrity is about security. Although security is certainly one of integrity?s impacts (albeit not in the way hypothesised in this discussion) other major impacts are on performance (including, though certainly not limited to, link-time optimisations that are of interest to Project Leyden) and code evolution (the lack of integrity has been, by far, the biggest cause of JDK upgrade issues experienced by many applications an libraries). > > - A second, related concern is that flipping the default for this configuration in an LTS release as the first exposure to it for most people is more likely to derail deployment plans for users than if the default were flipped in a non-LTS release. If this change were deferred to jdk22 then that would give those planning deployment on (or upgrade to) jdk25 and also those planning to upgrade from jdk17 to jdk21 more time to discover and respond to the change. While we should certainly discuss timing after publishing the draft JEP, I?m not sure how relevant this particular argument is. Those who upgrade from 17 to 21 don?t care which of the versions they skipped introduced a change, and even the deprecation process does not take into account versions for which Oracle and other vendors choose to offer an LTS service. JDK 17 itself also introduced a far bigger tightening of strong encapsulation than the one discussed here. Furthermore, those who wish to upgrade from one version that has LTS offerings to another avail themselves of the LTS service to upgrade not immediately when the new version is released, so they are under no time pressure. > > - A third concern, already pointed out by Volker, is that some users may run their Java apps via launcher apps or scripts that mask access to the Java command line. For such users the change of default may mean that they lose the option to deploy dynamic agents for important ancillary tasks such as observability. We are not clear how many of our users this affects but we will be looking into this and hope to bring feedback to the JEP review. > Obviously, this problem can be remedied relatively easily by the supplier of the launcher enabling agent use or providing a suitable control switch. Our concern is not with how to solve this problem rather how the involvement of two parties, supplier and end user, might imply a need for the JEP to be targeted to a later release. This, too, is an argument that?s hard for me to understand. First, many JDK releases require changes to the command line, for various reasons. JDK 17 required bigger changes than the one announced here, and JDK 21 itself may well require other such changes that impact even more applications than this one ? making it an opportunity rather than a liability. Second, such changes are normally announced *later* than this one has. If an application under such constraints always uses the current JDK release, then surely a six-month notice is enough, and if it opts to use an LTS service, then it?s under no pressure. Anyway, let?s continue this discussion after I publish the draft JEP. ? Ron From ron.pressler at oracle.com Mon Mar 27 10:46:58 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Mon, 27 Mar 2023 10:46:58 +0000 Subject: [External] : Re: Disallowing the dynamic loading of agents by default In-Reply-To: <39D27A60-7914-4109-A19E-96AB5608A4F5@cox.net> References: <8DB1B340-1DBD-5A46-BE6C-24364031F5B4@hxcore.ol> <39D27A60-7914-4109-A19E-96AB5608A4F5@cox.net> Message-ID: <67A419ED-7C7B-4548-BD2A-922AFC9422B2@oracle.com> > On 25 Mar 2023, at 19:56, Gregg G Wonderly wrote: > > I understand you may have personal experiences with how you use Java. In my experience and others, Java has constantly had fundamental breakage in various details due to lack of understanding, on the platform development team(s) of what people actually do with Java. > > Sent from my iPhone Since at this point it is an established fact that the vast majority of JDK upgrade difficulties (certainly weighed by cost) have been the result of lack of integrity guarantees made by strong encapsulation, those who care about stability should welcome this change and the others to follow that will close the remaining loopholes in strong encapsulation. Strong encapsulation has already proven itself over the past several releases in allowing us to make big internal changes to the platform (such as those required for virtual threads) with little adverse effect on compatibility, something that wasn?t possible before. The success of strong encapsulation in that area shows that it?s the right approach for letting Java evolve, even in very significant ways, and at the same time reduce upgrade costs. ? Ron From volker.simonis at gmail.com Mon Mar 27 12:37:48 2023 From: volker.simonis at gmail.com (Volker Simonis) Date: Mon, 27 Mar 2023 14:37:48 +0200 Subject: [External] : Re: Disallowing the dynamic loading of agents by default In-Reply-To: <023F29ED-CAF3-4D32-B36C-8053DDCC580A@oracle.com> References: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> <023F29ED-CAF3-4D32-B36C-8053DDCC580A@oracle.com> Message-ID: On Mon, Mar 27, 2023 at 12:32?PM Ron Pressler wrote: > > Hi Andrew! > > > On 24 Mar 2023, at 17:21, Andrew Dinn wrote: > > > > Hi Ron, > > > > Thank you for providing a heads up on the proposed JEP. The Red Hat Java team have been discussing this proposal. We have reviewed the original discussion and also the surrounding debate which established requirements for adaptation of Jigsaw to incorporate the needs of agents. > > > > As an aside, I'll note that a thorough review was necessary /even/ in my case, despite the fact that I was an active party, because the discussion occurred, and corresponding decisions were made, quite some time ago. I mention this because it may explain the air of surprise and the desire to reiterate some of the original debate on the part of some respondents in this thread, who perhaps were not party, or only tangentially party, to the discussion. > > > > That also suggests that there may be a lot users who are not aware that the -XX:+EnableDynamicAgentLoading switch exists or do not really understand why it exists i.e. that there is a broad education issue at play here. > > Understood. > > > > > We do have some concerns about the JEP, specifically about the timing of its delivery. These are probably best addressed via the normal review process. In particular that will ensure the discussion happens in a more suitable and more widely subscribed forum than the Jigsaw list. However, I will briefly mention our concerns in this reply. Before that let me start with a few disclaimers: > > > > - We acknowledge that there is little to be gained from re-iterating arguments made in the previous discussion (although that does not imply the JEP review would not benefit from new arguments, especially from those who were not involved in that discussion) > > > > - We recognize that the purpose of the -XX:+EnableDynamicAgentLoading switch is to offer a platform integrity guarantee and that this change of the default reflects a desire to prioritise integrity over the flexibility that agents provide > > I would qualify that: we want to prioritise integrity *by default*. Integrity is only practical when it is the default, as adding more integrity rules after the fact is hard to the point of being impractical. Agents in themselves don?t impact that default, but dynamically loaded agents do. > > > > > - We recognize that the proposal is only proposing to flip a configuration default rather than detract from (or modify) available functionality > > > > - We recognize that changing this default will still allow (*most*) users to configure the behaviour they desire > > > > - We recognize that this advance notice has been given precisely to ensure that anyone wishing to deploy on jdk21 an app that relies on use of agents has time to plan appropriate configuration for their deployment > > > > - We recognize that this change of default is not being proposed for backport and hence that it will largely only affect the relatively small number of users who are currently developing for jdk21+ > > > > So, given that as a base for our comments where is the beef? > > > > - Our main concern is, predictably, timing. Clearly, this is a future, potential problem rather than a present problem - no one can be deploying on jdk21 yet and most developers who are currently preparing an app for deployment on jdk21+ will likely encounter the effect of this change before actual deployment and be in a position to remedy it. The concern is that advertising a change like this and getting users prepared to respond to it has always been difficult to achieve. In particular we expect a long tail of support problems from users who are trying to upgrade deployments from earlier releases to jdk21. > > So, while it is nice to have such early notice of the proposal we plan to review its likely impact on our users and how much time we need to prepare ourselves and our users to negotiate this change in behaviour. Any evidence we obtain to suggest a delay in targeting is appropriate will be brought to the JEP review. > > Very well, we can certainly discuss timing as part of the JEP discussion. > > The JEP itself is rather ambitious because I noticed that most of the strong encapsulation JEPs lacked a substantial motivation section (largely because they had been written before that became the norm) and so strong encapsulation has not been motivated in JEP form, and this is an opportunity to start rectifying that. This is important because some, even in this discussion, are under the impression that integrity is about security. Although security is certainly one of integrity?s impacts (albeit not in the way hypothesised in this discussion) other major impacts are on performance (including, though certainly not limited to, link-time optimisations that are of interest to Project Leyden) and code evolution (the lack of integrity has been, by far, the biggest cause of JDK upgrade issues experienced by many applications an libraries). Thanks for pointing out that "integritiy" and "security" are two different things and that this discussion is mostly about other aspects of "integrity" like performance and code evaluation. This is actually exactly why I already tried to ask several times about the "real" and/or the long term background of this change. Currently the JEP only seems to propose the change of the default value for dynamic agent loading. It is obviously not hard for other JDK vendors to use a different default and I agree that it is probably still manageable (though inconvenient) for administrators/operators to change the default at launch time. BUT, you rightly mention that once higher integrity is the DEFAULT, this opens the door for future optimizations (you listed some of them) and even completely different execution models / semantics for Java applications (e.g. as explored by Project Leyden). Once such new optimizations will be in place (and only work for the default, disabled dynamic agent loading setting) it will be much harder for users who depend on dynamic loading to enable it, because it will either impact their performance or it will limit their ability to use certain platform features. My main concern with the proposed change is not the current proposal but the impact it will have on the evolution of Java. Java's dynamic features are one of its biggest strength and a major reason for its success. Sacrificing some of them or making their usage increasingly expensive requires a broader discussion in the community and shouldn't happen "under the hood". I'm happy to continue that discussion on the actual JEP proposal. > > > > > - A second, related concern is that flipping the default for this configuration in an LTS release as the first exposure to it for most people is more likely to derail deployment plans for users than if the default were flipped in a non-LTS release. If this change were deferred to jdk22 then that would give those planning deployment on (or upgrade to) jdk25 and also those planning to upgrade from jdk17 to jdk21 more time to discover and respond to the change. > > While we should certainly discuss timing after publishing the draft JEP, I?m not sure how relevant this particular argument is. Those who upgrade from 17 to 21 don?t care which of the versions they skipped introduced a change, and even the deprecation process does not take into account versions for which Oracle and other vendors choose to offer an LTS service. JDK 17 itself also introduced a far bigger tightening of strong encapsulation than the one discussed here. Furthermore, those who wish to upgrade from one version that has LTS offerings to another avail themselves of the LTS service to upgrade not immediately when the new version is released, so they are under no time pressure. > > > > > - A third concern, already pointed out by Volker, is that some users may run their Java apps via launcher apps or scripts that mask access to the Java command line. For such users the change of default may mean that they lose the option to deploy dynamic agents for important ancillary tasks such as observability. We are not clear how many of our users this affects but we will be looking into this and hope to bring feedback to the JEP review. > > Obviously, this problem can be remedied relatively easily by the supplier of the launcher enabling agent use or providing a suitable control switch. Our concern is not with how to solve this problem rather how the involvement of two parties, supplier and end user, might imply a need for the JEP to be targeted to a later release. > > This, too, is an argument that?s hard for me to understand. First, many JDK releases require changes to the command line, for various reasons. JDK 17 required bigger changes than the one announced here, and JDK 21 itself may well require other such changes that impact even more applications than this one ? making it an opportunity rather than a liability. Second, such changes are normally announced *later* than this one has. If an application under such constraints always uses the current JDK release, then surely a six-month notice is enough, and if it opts to use an LTS service, then it?s under no pressure. > > Anyway, let?s continue this discussion after I publish the draft JEP. > > ? Ron > > From adinn at redhat.com Mon Mar 27 16:06:32 2023 From: adinn at redhat.com (Andrew Dinn) Date: Mon, 27 Mar 2023 17:06:32 +0100 Subject: [External] : Re: Disallowing the dynamic loading of agents by default In-Reply-To: References: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> <023F29ED-CAF3-4D32-B36C-8053DDCC580A@oracle.com> Message-ID: <6aba7a4c-acc0-8026-e35e-e123200dab1c@redhat.com> On 27/03/2023 13:37, Volker Simonis wrote: >> The JEP itself is rather ambitious because I noticed that most of the strong encapsulation JEPs lacked a substantial motivation section (largely because they had been written before that became the norm) and so strong encapsulation has not been motivated in JEP form, and this is an opportunity to start rectifying that. This is important because some, even in this discussion, are under the impression that integrity is about security. Although security is certainly one of integrity?s impacts (albeit not in the way hypothesised in this discussion) other major impacts are on performance (including, though certainly not limited to, link-time optimisations that are of interest to Project Leyden) and code evolution (the lack of integrity has been, by far, the biggest cause of JDK upgrade issues experienced by many applications an libraries). > > Thanks for pointing out that "integritiy" and "security" are two > different things and that this discussion is mostly about other > aspects of "integrity" like performance and code evaluation. This is > actually exactly why I already tried to ask several times about the > "real" and/or the long term background of this change. Currently the > JEP only seems to propose the change of the default value for dynamic > agent loading. It is obviously not hard for other JDK vendors to use a > different default and I agree that it is probably still manageable > (though inconvenient) for administrators/operators to change the > default at launch time. BUT, you rightly mention that once higher > integrity is the DEFAULT, this opens the door for future optimizations > (you listed some of them) and even completely different execution > models / semantics for Java applications (e.g. as explored by Project > Leyden). Once such new optimizations will be in place (and only work > for the default, disabled dynamic agent loading setting) it will be > much harder for users who depend on dynamic loading to enable it, > because it will either impact their performance or it will limit their > ability to use certain platform features. Red Hat's team had discusssed this issue and we did think about raising it in the previous reply. It is perhaps better left for the JEP discussion but we certainly consider it as important an issue as Volker does. Personally, I will note that this has been point I have been concerned by since the associated dilemma was raised by Mark Reinhold during the original discussion. Indeed, the possibility that retaining agents might necessitate a divergence in JVM (or even JDK) behaviour from the status quo when no agent is, nor can be, installed struck me very hard. My take-away was not an immediate abreaction. Rather: the opportunity agents provide for dynamic adaptation of the runtime is provisional and, needs to be balanced against the benefits that might accrue from them not being the picture; in the longer term some of the capabilities provided through the use of agents may be unsustainable. As an example of that I considered my agent, Byteman. It is immensely useful for injecting faults, validity assertions and monitoring capabilities into app code during unit system testing, avoiding the need for that code to actually appear in the product. The precise targeting of Byteman rules to specific code locations means that the injected code only minimally perturbs the whole code base. Notably, Byteman does not support bulk (online) transformation. Too much transformed code means you are effectively testing a very different app to the one you deploy, at least as far as execution speed, timings, resource use etc are concerned, and possibly even because of too radical a change to the app semantics. Clearly, if simply loading an agent provides opportunities may eventually lead to for the runtime being able to operate more efficiently and, as a result, cause in a significant change in execution speed, timings, resource use then a tool like Byteman becomes much less useful for this sort of testing. So, for me the writing was already on the wall back in jdk9 time. We may well have to trade off of some build/test time benefits against the undeniable impetus to improve deploy-time performance. > My main concern with the proposed change is not the current proposal > but the impact it will have on the evolution of Java. Java's dynamic > features are one of its biggest strength and a major reason for its > success. Sacrificing some of them or making their usage increasingly > expensive requires a broader discussion in the community and shouldn't > happen "under the hood". I'm happy to continue that discussion on the > actual JEP proposal. Amen to that. regards, Andrew Dinn ----------- From adinn at redhat.com Mon Mar 27 16:10:47 2023 From: adinn at redhat.com (Andrew Dinn) Date: Mon, 27 Mar 2023 17:10:47 +0100 Subject: [External] : Re: Disallowing the dynamic loading of agents by default In-Reply-To: <6aba7a4c-acc0-8026-e35e-e123200dab1c@redhat.com> References: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> <023F29ED-CAF3-4D32-B36C-8053DDCC580A@oracle.com> <6aba7a4c-acc0-8026-e35e-e123200dab1c@redhat.com> Message-ID: <862f5601-9930-2930-5b67-429537f75ba3@redhat.com> Sorry, let me correct some of my mangled grammar On 27/03/2023 17:06, Andrew Dinn wrote: > On 27/03/2023 13:37, Volker Simonis wrote: >>> The JEP itself is rather ambitious because I noticed that most of the >>> strong encapsulation JEPs lacked a substantial motivation section >>> (largely because they had been written before that became the norm) >>> and so strong encapsulation has not been motivated in JEP form, and >>> this is an opportunity to start rectifying that. This is important >>> because some, even in this discussion, are under the impression that >>> integrity is about security. Although security is certainly one of >>> integrity?s impacts (albeit not in the way hypothesised in this >>> discussion) other major impacts are on performance (including, though >>> certainly not limited to, link-time optimisations that are of >>> interest to Project Leyden) and code evolution (the lack of integrity >>> has been, by far, the biggest cause of JDK upgrade issues experienced >>> by many applications an libraries). >> >> Thanks for pointing out that "integritiy" and "security" are two >> different things and that this discussion is mostly about other >> aspects of "integrity" like performance and code evaluation. This is >> actually exactly why I already tried to ask several times about the >> "real" and/or the long term background of this change. Currently the >> JEP only seems to propose the change of the default value for dynamic >> agent loading. It is obviously not hard for other JDK vendors to use a >> different default and I agree that it is probably still manageable >> (though inconvenient) for administrators/operators to change the >> default at launch time. BUT, you rightly mention that once higher >> integrity is the DEFAULT, this opens the door for future optimizations >> (you listed some of them) and even completely different execution >> models / semantics for Java applications (e.g. as explored by Project >> Leyden). Once such new optimizations will be in place (and only work >> for the default, disabled dynamic agent loading setting) it will be >> much harder for users who depend on dynamic loading to enable it, >> because it will either impact their performance or it will limit their >> ability to use certain platform features. > > Red Hat's team had discusssed this issue and we did think about raising > it in the previous reply. It is perhaps better left for the JEP > discussion but we certainly consider it as important an issue as Volker > does. > > Personally, I will note that this is a point I have been concerned > by since the associated dilemma was raised by Mark Reinhold during the > original discussion. Indeed, the possibility that retaining agents might > necessitate a divergence in JVM (or even JDK) behaviour from the status > quo when no agent is, nor can be, installed struck me very hard. My > take-away was not an immediate abreaction. Rather: the opportunity > agents provide for dynamic adaptation of the runtime is provisional and, > needs to be balanced against the benefits that might accrue from them > not being the picture; in the longer term some of the capabilities > provided through the use of agents may be unsustainable. > > As an example of that I considered my agent, Byteman. It is immensely > useful for injecting faults, validity assertions and monitoring > capabilities into app code during unit system testing, avoiding the need > for that code to actually appear in the product. The precise targeting > of Byteman rules to specific code locations means that the injected code > only minimally perturbs the whole code base. > > Notably, Byteman does not support bulk (online) transformation. Too much > transformed code means you are effectively testing a very different app > to the one you deploy, at least as far as execution speed, timings, > resource use etc are concerned, and possibly even because of too radical > a change to the app semantics. > > Clearly, if simply loading an agent removes opportunities > for the runtime to operate more > efficiently and, as a result, causes a significant change in execution > speed, timings, resource use then a tool like Byteman becomes much less > useful for this sort of testing. So, for me the writing was already on > the wall back in jdk9 time. We may well have to trade off of some > build/test time benefits against the undeniable impetus to improve > deploy-time performance. > >> My main concern with the proposed change is not the current proposal >> but the impact it will have on the evolution of Java. Java's dynamic >> features are one of its biggest strength and a major reason for its >> success. Sacrificing some of them or making their usage increasingly >> expensive requires a broader discussion in the community and shouldn't >> happen "under the hood". I'm happy to continue that discussion on the >> actual JEP proposal. > Amen to that. > > regards, > > > Andrew Dinn > ----------- > -- regards, Andrew Dinn ----------- Red Hat Distinguished Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill From adinn at redhat.com Mon Mar 27 16:30:41 2023 From: adinn at redhat.com (Andrew Dinn) Date: Mon, 27 Mar 2023 17:30:41 +0100 Subject: [External] : Re: Disallowing the dynamic loading of agents by default In-Reply-To: <023F29ED-CAF3-4D32-B36C-8053DDCC580A@oracle.com> References: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> <023F29ED-CAF3-4D32-B36C-8053DDCC580A@oracle.com> Message-ID: <3e96a5df-c8b0-3574-a98b-33668391f3f0@redhat.com> Hi Ron, Thanks for the reply, I believe we have established a lot of common ground here. I'll try to clarify a couple of the things you found difficult to follow. On 27/03/2023 11:32, Ron Pressler wrote: >> - A second, related concern is that flipping the default for this configuration in an LTS release as the first exposure to it for most people is more likely to derail deployment plans for users than if the default were flipped in a non-LTS release. If this change were deferred to jdk22 then that would give those planning deployment on (or upgrade to) jdk25 and also those planning to upgrade from jdk17 to jdk21 more time to discover and respond to the change. > > While we should certainly discuss timing after publishing the draft JEP, I?m not sure how relevant this particular argument is. Those who upgrade from 17 to 21 don?t care which of the versions they skipped introduced a change, and even the deprecation process does not take into account versions for which Oracle and other vendors choose to offer an LTS service. JDK 17 itself also introduced a far bigger tightening of strong encapsulation than the one discussed here. Furthermore, those who wish to upgrade from one version that has LTS offerings to another avail themselves of the LTS service to upgrade not immediately when the new version is released, so they are under no time pressure. It seems to me to be a fairly simple point but I obviously didn't express it very well. Here's another try. If this is pushed in jdk21 then anyone currently developing or upgrading an app to target jdk21 will only have been able to test on jdk17-jdk20 where they will not encounter the issue. So, his nly leaves them a small window to detect that there will be a problem using agents in jdk21. When jdk21 arrives this may force them to delay deployment or they may even deploy unaware that the problem exists. If this is pushed in jdk22 instead of jdk21 then anyone who upgrades from jdk17 to jdk21 will not have a problem. Anyone working on an app for deployment on jdk25 will have the opportunity to test on 3 non_LTS releases which might manifest the potential agent problem before deployment. I hope that explains the problem better. >> - A third concern, already pointed out by Volker, is that some users may run their Java apps via launcher apps or scripts that mask access to the Java command line. For such users the change of default may mean that they lose the option to deploy dynamic agents for important ancillary tasks such as observability. We are not clear how many of our users this affects but we will be looking into this and hope to bring feedback to the JEP review. >> Obviously, this problem can be remedied relatively easily by the supplier of the launcher enabling agent use or providing a suitable control switch. Our concern is not with how to solve this problem rather how the involvement of two parties, supplier and end user, might imply a need for the JEP to be targeted to a later release. > > This, too, is an argument that?s hard for me to understand. First, many JDK releases require changes to the command line, for various reasons. JDK 17 required bigger changes than the one announced here, and JDK 21 itself may well require other such changes that impact even more applications than this one ? making it an opportunity rather than a liability. Second, such changes are normally announced *later* than this one has. If an application under such constraints always uses the current JDK release, then surely a six-month notice is enough, and if it opts to use an LTS service, then it?s under no pressure. Of course, I accept that changes to command line options are nothing new. However, I don't quite see how to get from there to the implication that this specific change cannot therefore raise concerns. I think the truth of that conclusion has to depend on details of the change, specifically what the effect might be on users. regards, Andrew Dinn ----------- Red Hat Distinguished Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill From greggwon at cox.net Tue Mar 28 03:34:02 2023 From: greggwon at cox.net (Gregg Wonderly) Date: Mon, 27 Mar 2023 22:34:02 -0500 Subject: [External] : Re: Disallowing the dynamic loading of agents by default In-Reply-To: <3e96a5df-c8b0-3574-a98b-33668391f3f0@redhat.com> References: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> <023F29ED-CAF3-4D32-B36C-8053DDCC580A@oracle.com> <3e96a5df-c8b0-3574-a98b-33668391f3f0@redhat.com> Message-ID: On Mar 27, 2023, at 11:30 AM, Andrew Dinn wrote: > If this is pushed in jdk21 then anyone currently developing or upgrading an app to target jdk21 will only have been able to test on jdk17-jdk20 where they will not encounter the issue. So, his nly leaves them a small window to detect that there will be a problem using agents in jdk21. When jdk21 arrives this may force them to delay deployment or they may even deploy unaware that the problem exists. > > If this is pushed in jdk22 instead of jdk21 then anyone who upgrades from jdk17 to jdk21 will not have a problem. Anyone working on an app for deployment on jdk25 will have the opportunity to test on 3 non_LTS releases which might manifest the potential agent problem before deployment. This is, again, where the reality that some Java users live in is very different than what seems to be known my many decision makers here. Most corporate users of Java don?t control when a particular version of Java is deployed into their environments. It keeps being proposed, that somehow users are deploying a specific version of Java and getting an appropriate version of an application they use, all at one moment in time. The supposition that Java is ?deployed? with a particular software system that uses it, is summarily false. Even Linux releases by Redhat for RHEL, Centos or even Fedora don?t let you pick any and every version of Java. Java applications, by the millions were written without needing any particular vision of Java, until a version broke something major like starting at Java2 (1.2) and then Java1.4 which need a dozen fixes and then Java 1.5 that broke huge numbers of desktop apps that did not that had not used volatile class values for so many things, including loop control values that kept loops from exiting. Then we had 1.9 that almost went out the door disabling every single Spring app in existence. And there are more and more things happening that just do not make much sense in the grand scheme of things. Overall Java is just not a safe place to take people for the first time. Many have had horrid problems and given up on Java. For Java1.2, Perforce invested huge time to try and create a new desktop app for their SCM system. They got to the point of almost releasing a beta to testers and then summarily threw it all away because they just could not make it work for all the things that got broke in 1.2. At my business, we have lots of each device applications where it would be a good thing to use, but because of the breakage and issues others have experience with Java over the years, their experiences cause them to just say no to anything Java. Java is just randomly upgraded on peoples desktops in their view. It gets replaced by the IT staff at most corps at unrelated moments that they start using a particular Java application. Those corps and their IT staff have little to no knowledge of what every Java application is let alone how it might be dependent on features that are being changed at each release of the JDK/JVM. The end result is that it is a surprise, always for this class of user, which version of Java will be available and which application will break this time. I still have lots of ?jar? file applications that I share with others and they just double click on those to run them. It?s that class of user that this Java upgrades happen with software updates/distributions process completely overlooks. The Java licensing was always about you could not use Java as the sole application platform on a computer. So, all kinds of ?free? desktop apps (and applets and Java Web Start) applications were created and used by literally thousands of users that are completely out of sight. I continue to see massive migration away from Java as the first choice for new applications amongst developers I talk to. It?s not being taught to most new developers I meet. They hardly even know that Java exists or what it?s capable of. Most developers seem to be taught web front end development tools or .net or golang or other languages besides Java for backend dev. There seems to be little chance that Java will have a place in the landscape within the next 5 years or so as those who have used Java since the mid 1990s age out of the pool of active developers and are no longer influencing tech used. What a sad tale the last 10 years of Java has been compared to what was possible 25 years ago, and should of happened? Gregg Wonderly From adinn at redhat.com Tue Mar 28 09:13:17 2023 From: adinn at redhat.com (Andrew Dinn) Date: Tue, 28 Mar 2023 10:13:17 +0100 Subject: [External] : Re: Disallowing the dynamic loading of agents by default In-Reply-To: References: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> <023F29ED-CAF3-4D32-B36C-8053DDCC580A@oracle.com> <3e96a5df-c8b0-3574-a98b-33668391f3f0@redhat.com> Message-ID: Greeg, I won't respond point by point to your comments as I cannot see any great value in doing so. I really only want to make one general comment about your account below, which is that you appear to me to be relaying your own experience as a desktop Java user and universalising it to all users and uses of Java. While I acknowledge that you are correct to state that there can be problems with maintaining a consistent desktop setup for Java I'll counter that with two important qualifying arguments which undercut your account. Firstly, the problems that you describe are general ones that apply when it comes to software configuration management for a highly user-specific and often, in consequence, highly variable environment like a desktop. They are not problems that are specific to Java or other language runtimes. Even within that category many runtimes, especially language runtimes, suffer from problems with installed version mismatches and in many cases these have been notably far worse problems than Java (as for example the Python 2/3 fiasco mentioned by Andrew Haley). However, that is not to say that the problem lies with the runtime itself. Version management problems cannot simply be resolved by preserving Java (or any other deployed software) in aspic. At least some minimal level of upgrade of a runtime like Java is needed to deal with emerging security and critical functional problems. However, in the longer term any platform will also need to incorporate larger scale modifications in order to cater for the continuous, dramatic change that we have seen and continue to see in all hardware and operating systems. Java has not stood still over the last 25 years for very good reasons. Your reply suggests that you are unaware of the reality that those who manage non-desktop deployments plan very carefully around this need to adapt deployments to updates. Your lament that (your and others') desktop management does not include such provision may reflect the reality of some (but definitely not all) individuals or organizations. However, that lack of provision attests not to any failing on the part of the developers of Java but rather to a lack of organization, understanding and adequate preparation for *necessary maintenance* on the part of those responsible for managing said desktops. Which brings us to the second point: your complaint omits to allow for the enormous efforts that Java developers perform to enable Java users to rely on and profit from exactly the sort of continuity that you misguidedly claim Java does not provide. We are currently maintaining reliable, secure and bug free versions of jdk8, jdk11, jdk17 which allow users to continue to run applications that were originally deployed many years ago and will do so for many years to come. Indeed, as with many other large-scale, organized open source software infrastructure projects, this is the primary focus of the OpenJDK team. The number of people involved in maintaining legacy releases of Java to support existing users far outweighs those involved in developing new releases and new features. Users who put in the work needed to manage the configuration of their desktop environments can easily use these legacy releases to maintain their own desktop applications. It's not a free lunch -- admins of the desktop systems need to have at least a moderate understanding of how to configure their systems in order to maintain applications that rely on a specific Java platform release. However, to claim that the OpenJDK devs have not made this possible, worse to claim that Java has actually poisoned the well for desktop users, is a ridiculous and ignorant assertion. regards, Andrew Dinn ----------- Red Hat Distinguished Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill On 28/03/2023 04:34, Gregg Wonderly wrote: > On Mar 27, 2023, at 11:30 AM, Andrew Dinn wrote: >> If this is pushed in jdk21 then anyone currently developing or upgrading an app to target jdk21 will only have been able to test on jdk17-jdk20 where they will not encounter the issue. So, his nly leaves them a small window to detect that there will be a problem using agents in jdk21. When jdk21 arrives this may force them to delay deployment or they may even deploy unaware that the problem exists. >> >> If this is pushed in jdk22 instead of jdk21 then anyone who upgrades from jdk17 to jdk21 will not have a problem. Anyone working on an app for deployment on jdk25 will have the opportunity to test on 3 non_LTS releases which might manifest the potential agent problem before deployment. > > This is, again, where the reality that some Java users live in is very different than what seems to be known my many decision makers here. Most corporate users of Java don?t control when a particular version of Java is deployed into their environments. It keeps being proposed, that somehow users are deploying a specific version of Java and getting an appropriate version of an application they use, all at one moment in time. The supposition that Java is ?deployed? with a particular software system that uses it, is summarily false. Even Linux releases by Redhat for RHEL, Centos or even Fedora don?t let you pick any and every version of Java. Java applications, by the millions were written without needing any particular vision of Java, until a version broke something major like starting at Java2 (1.2) and then Java1.4 which need a dozen fixes and then Java 1.5 that broke huge numbers of desktop apps that did not that had not used volatile class values for so many things, including loop control values that kept loops from exiting. Then we had 1.9 that almost went out the door disabling every single Spring app in existence. And there are more and more things happening that just do not make much sense in the grand scheme of things. > > Overall Java is just not a safe place to take people for the first time. Many have had horrid problems and given up on Java. For Java1.2, Perforce invested huge time to try and create a new desktop app for their SCM system. They got to the point of almost releasing a beta to testers and then summarily threw it all away because they just could not make it work for all the things that got broke in 1.2. At my business, we have lots of each device applications where it would be a good thing to use, but because of the breakage and issues others have experience with Java over the years, their experiences cause them to just say no to anything Java. > > Java is just randomly upgraded on peoples desktops in their view. It gets replaced by the IT staff at most corps at unrelated moments that they start using a particular Java application. Those corps and their IT staff have little to no knowledge of what every Java application is let alone how it might be dependent on features that are being changed at each release of the JDK/JVM. > > The end result is that it is a surprise, always for this class of user, which version of Java will be available and which application will break this time. > > I still have lots of ?jar? file applications that I share with others and they just double click on those to run them. It?s that class of user that this Java upgrades happen with software updates/distributions process completely overlooks. The Java licensing was always about you could not use Java as the sole application platform on a computer. So, all kinds of ?free? desktop apps (and applets and Java Web Start) applications were created and used by literally thousands of users that are completely out of sight. > > I continue to see massive migration away from Java as the first choice for new applications amongst developers I talk to. It?s not being taught to most new developers I meet. They hardly even know that Java exists or what it?s capable of. Most developers seem to be taught web front end development tools or .net or golang or other languages besides Java for backend dev. > > There seems to be little chance that Java will have a place in the landscape within the next 5 years or so as those who have used Java since the mid 1990s age out of the pool of active developers and are no longer influencing tech used. > > What a sad tale the last 10 years of Java has been compared to what was possible 25 years ago, and should of happened? > > Gregg Wonderly > From ron.pressler at oracle.com Tue Mar 28 11:43:54 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Tue, 28 Mar 2023 11:43:54 +0000 Subject: [External] : Re: Disallowing the dynamic loading of agents by default In-Reply-To: References: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> <023F29ED-CAF3-4D32-B36C-8053DDCC580A@oracle.com> <3e96a5df-c8b0-3574-a98b-33668391f3f0@redhat.com> Message-ID: <1961D75D-1A31-4B10-A84B-72A9ADB065A6@oracle.com> Applications can now control the Java version available to them (and this is something we?ll keep improving), and the JRE, the centralised Java environment installed on user machines, has not existed for some time. Strong encapsulation (which this change is part of) has made compatibility better than it?s ever been. It is true that we cannot now fix the compatibility issues between Java 1.2 and 1.5, but rather than the reality not being known to decision makers, it looks like you may be unfamiliar with the changes made by decision makers (a few years ago) to address some of the very issued you raised. ? Ron > On 28 Mar 2023, at 04:34, Gregg Wonderly wrote: > > On Mar 27, 2023, at 11:30 AM, Andrew Dinn wrote: >> If this is pushed in jdk21 then anyone currently developing or upgrading an app to target jdk21 will only have been able to test on jdk17-jdk20 where they will not encounter the issue. So, his nly leaves them a small window to detect that there will be a problem using agents in jdk21. When jdk21 arrives this may force them to delay deployment or they may even deploy unaware that the problem exists. >> >> If this is pushed in jdk22 instead of jdk21 then anyone who upgrades from jdk17 to jdk21 will not have a problem. Anyone working on an app for deployment on jdk25 will have the opportunity to test on 3 non_LTS releases which might manifest the potential agent problem before deployment. > > This is, again, where the reality that some Java users live in is very different than what seems to be known my many decision makers here. Most corporate users of Java don?t control when a particular version of Java is deployed into their environments. It keeps being proposed, that somehow users are deploying a specific version of Java and getting an appropriate version of an application they use, all at one moment in time. The supposition that Java is ?deployed? with a particular software system that uses it, is summarily false. Even Linux releases by Redhat for RHEL, Centos or even Fedora don?t let you pick any and every version of Java. Java applications, by the millions were written without needing any particular vision of Java, until a version broke something major like starting at Java2 (1.2) and then Java1.4 which need a dozen fixes and then Java 1.5 that broke huge numbers of desktop apps that did not that had not used volatile class values for so many things, including loop control values that kept loops from exiting. Then we had 1.9 that almost went out the door disabling every single Spring app in existence. And there are more and more things happening that just do not make much sense in the grand scheme of things. > > Overall Java is just not a safe place to take people for the first time. Many have had horrid problems and given up on Java. For Java1.2, Perforce invested huge time to try and create a new desktop app for their SCM system. They got to the point of almost releasing a beta to testers and then summarily threw it all away because they just could not make it work for all the things that got broke in 1.2. At my business, we have lots of each device applications where it would be a good thing to use, but because of the breakage and issues others have experience with Java over the years, their experiences cause them to just say no to anything Java. > > Java is just randomly upgraded on peoples desktops in their view. It gets replaced by the IT staff at most corps at unrelated moments that they start using a particular Java application. Those corps and their IT staff have little to no knowledge of what every Java application is let alone how it might be dependent on features that are being changed at each release of the JDK/JVM. > > The end result is that it is a surprise, always for this class of user, which version of Java will be available and which application will break this time. > > I still have lots of ?jar? file applications that I share with others and they just double click on those to run them. It?s that class of user that this Java upgrades happen with software updates/distributions process completely overlooks. The Java licensing was always about you could not use Java as the sole application platform on a computer. So, all kinds of ?free? desktop apps (and applets and Java Web Start) applications were created and used by literally thousands of users that are completely out of sight. > > I continue to see massive migration away from Java as the first choice for new applications amongst developers I talk to. It?s not being taught to most new developers I meet. They hardly even know that Java exists or what it?s capable of. Most developers seem to be taught web front end development tools or .net or golang or other languages besides Java for backend dev. > > There seems to be little chance that Java will have a place in the landscape within the next 5 years or so as those who have used Java since the mid 1990s age out of the pool of active developers and are no longer influencing tech used. > > What a sad tale the last 10 years of Java has been compared to what was possible 25 years ago, and should of happened? > > Gregg Wonderly From greggwon at cox.net Tue Mar 28 15:35:41 2023 From: greggwon at cox.net (Gregg Wonderly) Date: Tue, 28 Mar 2023 10:35:41 -0500 Subject: [External] : Re: Disallowing the dynamic loading of agents by default In-Reply-To: References: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> <023F29ED-CAF3-4D32-B36C-8053DDCC580A@oracle.com> <3e96a5df-c8b0-3574-a98b-33668391f3f0@redhat.com> Message-ID: <041E6D30-C453-4C6C-8562-BC6545E032B1@cox.net> > On Mar 28, 2023, at 4:13 AM, Andrew Dinn wrote: > > Greeg, > > I won't respond point by point to your comments as I cannot see any great value in doing so. I really only want to make one general comment about your account below, which is that you appear to me to be relaying your own experience as a desktop Java user and universalising it to all users and uses of Java. While I acknowledge that you are correct to state that there can be problems with maintaining a consistent desktop setup for Java I'll counter that with two important qualifying arguments which undercut your account. Again, the supposition is that somehow users of software systems are always surrounded by version planning and management. This is just not the case. People use software for its functionality not because the platform has certain features. The platform is what enables software systems to be constructed and used. When the platform changes it?s feature set or design and removes features or disables the default use of those features, that lack of backward compatibility, when unmanaged by the platform, creates havoc for users and software developers who are providing software systems in vertical markets or into communities of users who are not educated or knowledgable enough to understand what any of these words actually refer to. > Firstly, the problems that you describe are general ones that apply when it comes to software configuration management for a highly user-specific and often, in consequence, highly variable environment like a desktop. They are not problems that are specific to Java or other language runtimes. Even within that category many runtimes, especially language runtimes, suffer from problems with installed version mismatches and in many cases these have been notably far worse problems than Java (as for example the Python 2/3 fiasco mentioned by Andrew Haley). However, that is not to say that the problem lies with the runtime itself. Your supposition is again that somehow everyone has the benefit of having a guru or appointed party that manages all of their software and versioning needs. I used the example of corporate users to draw on that environments ?random? upgrade timelines. Users get new machines as something fails or get new versions of software based on ?bugs? or ?risks? or other drivers that are completely unrelated to software development timelines of third party suppliers of software systems. Suggesting that just because this has happened historically, that somehow it provides the relief to do it as well is the point I am trying to counter with my arguments here. > Version management problems cannot simply be resolved by preserving Java (or any other deployed software) in aspic. At least some minimal level of upgrade of a runtime like Java is needed to deal with emerging security and critical functional problems. However, in the longer term any platform will also need to incorporate larger scale modifications in order to cater for the continuous, dramatic change that we have seen and continue to see in all hardware and operating systems. Java has not stood still over the last 25 years for very good reasons. Removing features and changing how the platform handles features such that software deployment or software systems have to change is the problem. There are so many things that have been changed to no real advantage to developers. Java 9 changes for ?modularity? were really only targeted changes to try and manage people plugging into private APIs, because the public APIs were not providing needed features and it was difficult or impossible to use the platform without those changes. This speaks to the ineffective nature of the platform management overall. Historically, and I assert this over and over, because I was in the room where it happened, Sun was only interested in selling servers. Oracle is again only interested in selling server/backend solutions and support. The OpenJDK community is, in fact providing a great service to the community in maintaining JDK versions that avoid forcing the user into incompatible places. > Your reply suggests that you are unaware of the reality that those who manage non-desktop deployments plan very carefully around this need to adapt deployments to updates. Your lament that (your and others') desktop management does not include such provision may reflect the reality of some (but definitely not all) individuals or organizations. However, that lack of provision attests not to any failing on the part of the developers of Java but rather to a lack of organization, understanding and adequate preparation for *necessary maintenance* on the part of those responsible for managing said desktops. I?ve done backend development with lots of different languages for decades. I?ve been subjected to Windows versioning, Linux Versioning (Redhat ripping out support for SATA2/SAS controllers in RHEL/Centos 8, cutting the chain for many servers that could easily move to 8, but would require huge customization of the boot to load the driver into the RAM disk) and lots of other versioning things. But, for a long time, I was on top of what was happening with Java because I was in the middle of things like the Jini community and the desktop communities. But, it became so painful and unreconcilable if not impossible to deal with the way that Java was being managed, that I gave up. I have 100,000s of thousands of lines of Java software that I?ve created all over the spectrum. Yet, its impossible for me to continue to be in the Java communities because I can?t get users interested in Java apps because they are absolutely put off by the version management issues and trying to understand which version of the runtime will work and exactly how they can manage to keep that in place in their environment. > Which brings us to the second point: your complaint omits to allow for the enormous efforts that Java developers perform to enable Java users to rely on and profit from exactly the sort of continuity that you misguidedly claim Java does not provide. We are currently maintaining reliable, secure and bug free versions of jdk8, jdk11, jdk17 which allow users to continue to run applications that were originally deployed many years ago and will do so for many years to come. Indeed, as with many other large-scale, organized open source software infrastructure projects, this is the primary focus of the OpenJDK team. The number of people involved in maintaining legacy releases of Java to support existing users far outweighs those involved in developing new releases and new features. I apologize for not including the work that the OpenJDK community does do. It?s quite remarkable that there is that dedication. I think that it reflects the same level of commitment I had to Java for some time. It would be a great platform and a great place to do lots of software that would be portable. However, the lack of attention to versioning across the portability landscape and the general problems created by continued change in platform behaviors that are not backward compatible are what create a huge problem for Java?s use in communities that are not commercial software systems. > Users who put in the work needed to manage the configuration of their desktop environments can easily use these legacy releases to maintain their own desktop applications. It's not a free lunch -- admins of the desktop systems need to have at least a moderate understanding of how to configure their systems in order to maintain applications that rely on a specific Java platform release. However, to claim that the OpenJDK devs have not made this possible, worse to claim that Java has actually poisoned the well for desktop users, is a ridiculous and ignorant assertion. Sure these legacy systems are available. But Java is not really helping the user with incompatible version changes. You have the problem that Java version installs only happen when people know what to install. You have the problems with non-elevated-privilege users can?t install new Java versions etc. There are just so many ways that these types of changes break Java?s write-once-run-anywhere promise and continue to disable existing systems without any details about what actually happened to break things. Gregg Wonderly From mike at plan99.net Tue Mar 28 16:11:15 2023 From: mike at plan99.net (Mike Hearn) Date: Tue, 28 Mar 2023 18:11:15 +0200 Subject: [External] : Re: Disallowing the dynamic loading of agents by default In-Reply-To: References: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> <023F29ED-CAF3-4D32-B36C-8053DDCC580A@oracle.com> <3e96a5df-c8b0-3574-a98b-33668391f3f0@redhat.com> Message-ID: Hi Gregg, Distributing little apps as JARs indeed doesn't work well anymore out of the box, but it doesn't have to be the end of the line for them. I've spent a couple of years writing a tool designed explicitly to solve all these problems [1]. You give Conveyor your JARs (or a Maven/Gradle build), it'll create and upload self-updating packages for Windows, Mac and Linux that bundle a jlinked and minified JVM, fully signed and notarized, along with a download HTML page for end users to get a big green button. It'll even draw an icon for you. You can do this from any OS, you don't need Windows or macOS to ship for them. This approach has the major downside that unless your app is open source it's not free (we gotta make money somehow) BUT if you can put that to one side, it works better than the JAR era ever could: - No Java compatibility issues by design. - Not blocked by browsers/operating system security. - Apps can update more smoothly than Web Start ever allowed. - You can use OS specific integrations. - Clean uninstalls, native code handled better and so on. You might object that this is somehow more effort than just making a fat jar and sending it to people, but in practice it's not harder. You run it, out pops a bunch of files, you make them available to people, done. W.R.T. corporate deployment, note that Conveyor makes MSIX files which are Microsoft's official format and easily deployed across Windows networks. The difficulty with the send-a-JAR approach is that maintaining backwards compatibility at the level you want (Win32, web level) takes a massive level of spend, a large library of public programs which can be automatically regression tested against, and a commitment to never break anything even if it seriously disadvantages later developers, and even then things will still break despite best intentions. Decades ago this tradeoff made more sense because bandwidth and storage space were much tighter, but now it's harder to justify. That's why so few platforms do it anymore. [1] https://hydraulic.software/ From greggwon at cox.net Wed Mar 29 00:29:59 2023 From: greggwon at cox.net (Gregg G Wonderly) Date: Tue, 28 Mar 2023 19:29:59 -0500 Subject: [External] : Re: Disallowing the dynamic loading of agents by default In-Reply-To: References: Message-ID: This is exactly my point! Why would any one want to do something like this? This level of workaround and specialized deployment is the kind of breakage that I am referring to. I just don?t understand how this kind of rigging and customization can even start to feel right. Gregg Wonderly > On Mar 28, 2023, at 11:11 AM, Mike Hearn wrote: > > ?Hi Gregg, > > Distributing little apps as JARs indeed doesn't work well anymore out > of the box, but it doesn't have to be the end of the line for them. > I've spent a couple of years writing a tool designed explicitly to > solve all these problems [1]. You give Conveyor your JARs (or a > Maven/Gradle build), it'll create and upload self-updating packages > for Windows, Mac and Linux that bundle a jlinked and minified JVM, > fully signed and notarized, along with a download HTML page for end > users to get a big green button. It'll even draw an icon for you. You > can do this from any OS, you don't need Windows or macOS to ship for > them. > > This approach has the major downside that unless your app is open > source it's not free (we gotta make money somehow) BUT if you can put > that to one side, it works better than the JAR era ever could: > > - No Java compatibility issues by design. > > - Not blocked by browsers/operating system security. > > - Apps can update more smoothly than Web Start ever allowed. > > - You can use OS specific integrations. > > - Clean uninstalls, native code handled better and so on. > > You might object that this is somehow more effort than just making a > fat jar and sending it to people, but in practice it's not harder. You > run it, out pops a bunch of files, you make them available to people, > done. > > W.R.T. corporate deployment, note that Conveyor makes MSIX files which > are Microsoft's official format and easily deployed across Windows > networks. > > The difficulty with the send-a-JAR approach is that maintaining > backwards compatibility at the level you want (Win32, web level) takes > a massive level of spend, a large library of public programs which can > be automatically regression tested against, and a commitment to never > break anything even if it seriously disadvantages later developers, > and even then things will still break despite best intentions. Decades > ago this tradeoff made more sense because bandwidth and storage space > were much tighter, but now it's harder to justify. That's why so few > platforms do it anymore. > > [1] https://hydraulic.software/ From mike at plan99.net Wed Mar 29 08:12:44 2023 From: mike at plan99.net (Mike Hearn) Date: Wed, 29 Mar 2023 10:12:44 +0200 Subject: [External] : Re: Disallowing the dynamic loading of agents by default In-Reply-To: References: Message-ID: Why, well, you get more features, it's easier for the end user, and not any harder for the developer. Those are pretty concrete reasons why people would want to do it that way. I'd suggest trying Conveyor out yourself before worrying about rigging or customization, because straightforward Java apps don't actually need any configuration beyond specifying the URL of the update site. It's easier than Web Start was, and takes no more lines of code than configuring Gradle to make fat JARs does. It's free for open source projects so I'd say please just try it out with an open mind. If you still find it harder than publishing JARs then I'd be very interested to read an essay or blog post drilling into the differences. Maybe try following the JavaFX tutorial here: https://conveyor.hydraulic.dev/7.2/tutorial/hare/jvm/ Without that kind of concrete detail though, I'm going to feel like this is about perspectives. If you view Java as a capital-p Platform, competing on the same level as an operating system, then JARs feel natural and bundling would feel like a retreat from the glory days. If you perceive it as a large and fancy library then it's just like any other library and asking the user to manage it separately makes no more sense than expecting users to manage the Visual C++ runtime. The switch to jlinking and bundling in this case is all win, it removes headaches instead of adding them. From ron.pressler at oracle.com Wed Mar 29 12:14:13 2023 From: ron.pressler at oracle.com (Ron Pressler) Date: Wed, 29 Mar 2023 12:14:13 +0000 Subject: [External] : Re: Disallowing the dynamic loading of agents by default In-Reply-To: References: Message-ID: > On 29 Mar 2023, at 01:29, Gregg G Wonderly wrote: > > This is exactly my point! Why would any one want to do something like this? This level of workaround and specialized deployment is the kind of breakage that I am referring to. I just don?t understand how this kind of rigging and customization can even start to feel right. > > Gregg Wonderly But you do understand, because you yourself have pointed out the problems caused by the old approach where the runtime and the application were provided separately. The current approach is a result of the JDK evolving to address those very problems, and it?s working. Embedded custom runtimes and strong encapsulation have greatly alleviated most of them. The alternative, a runtime that never changes, is only workable for very limited applications. The old approach was guided by one primary use case, Applets, which had very limited capabilities. Indeed, similarly restricted JavaScript applications are delivered separately from their runtime, the web browser, but the more capable desktop applications written in JavaScript are delivered with an embedded runtime, for similar reasons. In other languages, including ?native? ones, the trend is to similarly statically link dependencies ? including even libc ? or to bundle them in a container. These aren?t workarounds, but the best means we have to date to deliver applications that are portable, capable, and evolvable, whether it feels right or not. Perhaps someday another approach will present itself. ? Ron From adinn at redhat.com Wed Mar 29 14:49:05 2023 From: adinn at redhat.com (Andrew Dinn) Date: Wed, 29 Mar 2023 15:49:05 +0100 Subject: [External] : Re: Disallowing the dynamic loading of agents by default In-Reply-To: <041E6D30-C453-4C6C-8562-BC6545E032B1@cox.net> References: <5840A302-AD72-4308-A064-CB89868784C1@oracle.com> <023F29ED-CAF3-4D32-B36C-8053DDCC580A@oracle.com> <3e96a5df-c8b0-3574-a98b-33668391f3f0@redhat.com> <041E6D30-C453-4C6C-8562-BC6545E032B1@cox.net> Message-ID: <7949681a-2f89-0964-1bff-67238389aaf0@redhat.com> Hi Gregg, Thanks for your reply. I only have one small point to make. On 28/03/2023 16:35, Gregg Wonderly wrote: > Again, the supposition is that somehow users of software systems are always surrounded by version planning and management. . . . No, actually, my supposition is that users of software systems *ought* to have measures in place to handle version planning and management. The fact that many users do not, especially desktop users, merely reflects a widespread misunderstanding of how complex a thing it is to run a computer. The majority of car drivers, not being mechanically minded and skilled, invariably expect (even if they do not welcome) the regular expense required to keep their vehicles on the road. What is more they can pick and choose from a large number of service garages who will perform the necessary maintenance. Sadly, this analogy with running a computer breaks down not at the point where expert remedy is needed but where it is sought and paid for. regards, Andrew Dinn ----------- Red Hat Distinguished Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill