Disallowing the dynamic loading of agents by default

[Alan Bateman]

On 03/04/2017 15:19, Andrew Dinn wrote:

> :
> Well, Byteman is definitely in the cross hairs by virtue of both counts
> if those are the criteria (but I still don't see Byteman as a problem
> that requires disruption :-)
>
> That said, I don't quite follow how your last statement relates to the
> proposed change. It reads to me as if EnableDynamicAgentLoading=true is
> intended to stop out of process loading of instrumentation agents like
> Byteman as well as foil code that hoists an agent into the current JVM
> process. Have I misread the proposed behaviour? Or have I read it right?
> (with the corollary, I assume, that here you are you merely setting out
> what you would prefer to implement in contrast to what you can actually
> achieve).
>
> Also, can you provide a reason why you are so agin agent-hoisting from
> within a JVM? Is there a reason why it is such a cardinal sin?
As Mark said, this discussion got off to a bad start. We might have to 
explain the issue again.

Java SE 9 / JDK 9 brings strong encapsulation. The access control for 
the Java Language and VM has been extended to modules so that modules 
that don't want their internals to be accessed from code outside the 
module can do so. None of the core modules want their internals to be 
accessed so none of the core modules are open or open any packages. A 
consequence of this is that code on the class path or module path 
doesn't get to break in these modules. This is really nice but it 
exposes a lot of technical debt in existing code (as we've seen in mails 
here over the last 18 months).

If libraries and applications can't break in then what about tools? 
Tools are special, they get the capability to instrument almost every 
class. As you know, their capabilities have been expanded in Java SE 9 / 
JDK 9 as all the tool APIs have been updated to support modules. This 
means they can instrument code in modules as well as modify modules to 
export or open packages to other modules.

Now bring the attach API and late binding agents into the picture. This 
is where things blur and where the problem arises. A library can use the 
attach mechanism to load an agent into the current VM and break into any 
module. It's much easier in JDK 9 compared to previous releases because 
the jdk.attach module is resolved by default. All it takes is someone to 
post a solution on stackoverflow that spins a sneaky agent to leak the 
Instrumentation object to the library. It's just too easy to "migrate" 
existing reflection hacks.

The attach mechanism was of course never intended to be used this way. 
It was meant for troubleshooting tools and profilers/similar to load 
agents into running VMs. Back in the JDK 6 then we did consider 
disallowing attaching to the current VM but didn't enforce it - one 
reason is that it's not hard to just fork a VM with tools.jar on the 
class path and connect back to the parent.

So that is the context for the discussion. We need to find a good way to 
put the Genie back in its bottle. It may be that we have to disable 
attaching to the current or ancestor VMs. We may have to prohibit the 
instrumentation of core modules by late binding agents. We may have to 
do some disabling of agent loading. Maybe a combination. Suggestions and 
proposals are of course welcome.

-Alan