JEP draft: Disallow the Dynamic Loading of Agents by Default

[Kirk Pepperdine]

Hi Dan,

I was initially concerned with this proposal and that it was being introduced into an LTS has only added to that. You are quite correct that it’s not what we wish to be true that should be driving us. So yes, LTS’s are no different than other releases except that isn’t what our users are saying as they jump from LTS to LTS.

While I fear my being concerned by a pending change falls from a tendency towards "resistance to change”, I also know that changes in the tooling chain historically have oversized effects on an organizations ability to migrate to the latest and greatest. I’d feel a lot better about things if only I had the data to help me temper my level of concern. To that end I’ve been speaking to different projects to understand the impact this change might have on them. The feedback from our biggest project that will be directly impacted by this change have (loosely) said, dynamic attach offers us a simple model to deliver monitoring but we could switch to direct attach. The problem here is that this project is a bolt on to many other applications and while this project could change, it’s not clear (for many of the reasons you’ve pointed out below) that it’s customers could. Complicating this is that I believe that only a handful of devs/devops/ops people know this in the works.

Finally, a precedent for delaying the taking away capabilities from users was established with the release of modularity in Java 9. As you may recall, Java 9 broke so many things that deploying the full on application of modularity was helpful. I would think that a similar approach here where the full on change was applied after the LTS release would also help adaptation.

Kind regards,
Kirk

PS, this email is targeted for serviceability but has been crossposted so that others not following that list will be aware of pending changes.


> On Apr 28, 2023, at 7:30 PM, Dan Heidinga <heidinga at redhat.com> wrote:
> 
> Hi Ron,
> 
> Thanks for writing up the JEP draft outlining the proposal to disallow dynamic loading of agents by default.  The Red Hat Java team has continued to discuss this proposal internally and with our stakeholders.
> 
> While there is a general agreement (or at least acceptance) with the overall direction of this proposal, we still see the concerns raised by Andrew [0] as needing to be addressed.
> 
> So let’s start with the high-order bit: timing.
> 
> The JEP draft states it “intends to adopt the proposal [to disable agents by default] six years later, in 2023, for JDK 21.”  We would like to see this targeted to JDK 22 instead as the change has not been widely evangelized yet and comes as a surprise to many, both to those actively developing OpenJDK and to those monitoring its development.
> 
> We owe it to the community to give this proposal enough bake time, especially given that JDK 21 is an LTS release.  Though the official position is that LTS releases are no different than any other release, the actions of JDK consumers make them special.  Users will be surprised by this change when they migrate from JDK 17 to 21.  If we delay till JDK 22, we give the ecosystem time to adapt to the change before the next LTS.
> 
> Additionally, users who have tested with the -XX:-EnableDynamicAgentLoading option will have false expectations if they validated their use of jcmd to load the agent as the behaviour was not correct prior to JDK 21 [1].
> 
> The next concern is one you call out in the draft that “Java's excellent serviceability has long been a source of pride for the platform.”  We agree!  
> 
> Java currently has an excellent, prime position in Observability capabilities. For better or for worse, there are many Observability tools that have relied on dynamic attach to provide the necessary monitoring for workloads
> 
> It’s important we give Java’s monitoring tools sufficient time to migrate comfortably without shocking the ecosystem by changing the default in an LTS.  By delaying the change till JDK 22, we give the ecosystem 2 years to migrate and to prepare users for the change.
> 
> Additionally, this provides the time for Java’s profiling tools to adapt as well.  And for the “ad-hoc troubleshooting” tools - like Byteman - to retrain their users.
> 
> Finally, while it’s easy to agree with the principle that “the application owner is given the final say on whether to allow dynamic loading of agents”, the argument can (and should!) be made that those application owners have made that final decision by deploying the libraries and tools that use dynamic attach.  A JVM command line argument is not the only way to express their consent for code to run on their system.
> 
> For many production uses, the reality is more complicated than a single “application owner”. Take a Kubernetes cluster for example.  
> 
> Who is the application owner when deploying to a Kubernetes cluster?  The dev team that develops the application?  The operations team that manages the cluster?  The monitoring team that monitors the applications? The Support team that troubleshoots issues with the deployment?  
> 
> We should be careful not to understate or misrepresent the complex web of “owners” that are currently able (and, for business reasons, need) to apply agents dynamically.  Downplaying the complexity many organizations experience when dealing with changes to command line options (as an example) weakens the argument for changing today’s status quo.
> 
> We also know that in many cases customers and users may not be in a position to modify startup scripts (e.g. even to add in an extra parameter) as to do so may invalidate support contracts, etc.
> 
> Dynamically attached agents have been a “superpower” of Java and their use has greatly benefited the ecosystem as they’ve helped service and support use cases that otherwise aren’t possible, as they helped propel Java to the forefront of the Observability tooling, and allowed many other useful libraries to be developed.
> 
> Let’s delay flipping the default until JDK 22 to give the breadth of the ecosystem time to absorb this change.
> 
> –Dan
>  
> 
> [0] https://mail.openjdk.org/pipermail/serviceability-dev/2023-March/047084.html <https://mail.openjdk.org/pipermail/serviceability-dev/2023-March/047084.html>
> [1] https://bugs.openjdk.org/browse/JDK-8304438 <https://bugs.openjdk.org/browse/JDK-8304438>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/serviceability-dev/attachments/20230429/7f9e43e5/attachment.htm>