Disallowing the dynamic loading of agents by default

[Alasdair Nottingham]

Hi,

I’m the lead for WebSphere Liberty at IBM. Liberty uses a java agent, and this proposal will affect us. Our Java Agent is used to update the bytecode of our classes to add in instrumentation for debug logging and performance monitoring. In general it is attached via -javaagent, which wont be affected by this proposal. However there is one case where we do a dynamic attach of this agent. There is an industry trend towards running the application server as an uber-jar using java -jar <jar.name>. To support this in Liberty (in common with other application servers) our main method extracts the app server to disk at startup before bootstrapping the server from extracted jar files. One of the jar files extracted in this way is our Java agent, so we use the attach API to attach it so we get monitoring and debug logging. I had been thinking of expanding this to use the attach api in preference to -javaagent because that gets rid of this spurious error message from the mac JVM:

objc[56755]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/bin/java (0x10f4464c0) and /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/libinstrument.dylib (0x110dbb4e0). One of the two will be used. Which one is undefined.

As I understand it from the emails the issue is that the agent API is now more powerful in Java 9 and can be used to break modularity, and therefore there is a desire to restrict access to it to preserve the integrity of the modularity system. 

I’d like to add my support to Andrew’s points. I think they are well made and cover my concerns with the proposals well. I’d like to add my thoughts as well though.

As I see it there are these 4 possible scenarios:

1) Java agent specified using -javaagent
2) Java agent dynamically added by same JVM
3) Java program uses a library that secretly dynamically adds a Java agent to do break modularity.
4) Java agent dynamically added using external tool

The proposal does nothing to prevent the first scenario 1, and breaks everything using scenario 2 and 4 in the hope of preventing scenario 3. As a Java developer I can do what I want via scenario 1, so I have the magic bullets to break modularity. The downside is it requires collusion with the application user. I wish to remove the need for that collusion since it makes my application easier to use, so I prefer scenario 2.

I agree with Andrew that scenario 3 is unlikely to be a serious issue. As a developer when I pick up libraries I have a good understanding of what they do and I either accept, or I do not the implications. Checking to see if a library attaches a Java agent is trivial since you just need to look for a manifest header. I’m going to know if a library does this.

Personally I don’t care about scenario 4, but it seems scenario 4 is being broken merely to prevent someone shelling out to a new JVM that calls the attach API to attach the agent. The effect of this seems significant and negative. The fact that the enablement of the local jmx connector was highlighted as something not to break indicates that this is understood, but I don’t think the local jmx connector should have special status in this regard.

As stated above I do care about scenario 2 because I need it. Disabling Java agents from being dynamically attached provides zero real protection because I would just have my uber-jar main method extract my server and then launch a new JVM using -javaagent. This is not good from a usability perspective, but it is certainly an option. The proposal appears to me to provide no protection for modularity, but makes my application less usable on Java 9, vs Java 8.  As has been mentioned on this thread the people setting these options are the people least capable of understanding and getting them right.

Mark did ask for alternative proposals, so other than the defer to a future release option, what about allowing a developer to put something in the manifest.mf to enable dynamic attachment from the same JVM? If there is a concern over all or nothing being too broad the manifest entry could list a hash of the agent jar so the developer has to know about the agent a-priori. I think this would solve scenario 3, but wouldn’t break scenario 2 or 4. It wouldn’t protect against the child JVM does the bad attach, but perhaps that can be solve with a less blunt instrument. 

Thanks
Alasdair
STSM - IBM WebSphere Liberty

> On Apr 3, 2017, at 12:44 PM, Andrew Dinn <adinn at redhat.com> wrote:
> 
> On 03/04/17 16:50, Alan Bateman wrote:
>>> Also, can you provide a reason why you are so agin agent-hoisting from
>>> within a JVM? Is there a reason why it is such a cardinal sin?
>> As Mark said, this discussion got off to a bad start. We might have to
>> explain the issue again.
> 
> Sure, only I don't think I have misunderstood the existing explanations
> (that was what I wanted to check). However, I am not clear that I am
> getting the same, consistent explanation each time.
> 
>> Java SE 9 / JDK 9 brings strong encapsulation. The access control for
>> the Java Language and VM has been extended to modules so that modules
>> that don't want their internals to be accessed from code outside the
>> module can do so. None of the core modules want their internals to be
>> accessed so none of the core modules are open or open any packages. A
>> consequence of this is that code on the class path or module path
>> doesn't get to break in these modules. This is really nice but it
>> exposes a lot of technical debt in existing code (as we've seen in mails
>> here over the last 18 months).
> 
> Yes, it is really nice in many ways. But it is also not necessarily what
> everyone wants.
> 
> One thing I am not clear on as regards that 'really nice' is whether
> anything in the JVM wants -- or even hopes -- to rely on module
> encapsulation never being broken in order to provide 'semantic'
> guarantees. That's different from relying on it to provide security
> guarantees. The sort of thing I am thinking about is, say, a module-wide
> global analysis in the JIT guaranteeing that a call argument will only
> ever be non-NULL, a positive int, or some such invariant that can fed
> into an optimization phase. I can understand how a switch to disable
> dynamic agent loading might be needed to underline that sort of guarantee.
> 
>> . . .
>> Now bring the attach API and late binding agents into the picture. This
>> is where things blur and where the problem arises. A library can use the
>> attach mechanism to load an agent into the current VM and break into any
>> module. It's much easier in JDK 9 compared to previous releases because
>> the jdk.attach module is resolved by default. All it takes is someone to
>> post a solution on stackoverflow that spins a sneaky agent to leak the
>> Instrumentation object to the library. It's just too easy to "migrate"
>> existing reflection hacks.
> 
> I think this is already very well known technology and the presence of
> the jdk.attach module in the runtime was never really much of a bar. The
> people who want to do this certainly don't have to look for sneaky
> solutions on stackoverflow. However, I still don't really see your point
> here. If people want to migrate existing reflection hacks then they can
> and will do so by switching off your proposed flag or adding the agent
> at startup.
> 
> Are you perhaps concerned that users might have their hand forced by
> providers of library jars or middleware who hoist an agent into the JVM
> behind their backs? I think that would be rather a patronising view to
> take of the vast majority of producers and consumers of libraries and/or
> middleware. My belief is that anyone who attempted to provide a library
> or framework (open source or not) that disabled (some or all) modules by
> stealth would very soon be found out. I also believe they would
> immediately lose all (or, at least, most) of their prospective business
> from serious, paying customers. After all, that sort of behaviour
> *would* be a major security issue.
> 
> As I mentioned in my reply to Mark it is critical for users to know
> exactly how any agent they load into their JVM is going to modify the
> access restrictions in place in the JVM so that they can be sue that use
> of the agent is safe (just as they need to know that any jars they place
> into the classpath are not going to do things like round down their
> costs and pocket the spare change in a bank account somwehere). Users
> don't just add stuff to their classpath without knowing what it does and
> why it is appropriate.
> 
> Also, that doesn't mean there will be no market for code which
> advertises that it is going to change access controls. If users know
> what they are getting and know what the consequences are then I am sure
> there are cases where they can safely decide whether or not to use a
> library that opens up module access. It's the idea that people are going
> to be hoodwinked here -- if indeed that is your idea -- that I find
> rather difficult to follow. We seem to have moved from dynamic agent
> loading being a security issue that didn't seem to have a real basis to
> a 'users cannot be entrusted to look after their own classpath' issue
> that I find equally as baseless. Is there more to what you are saying
> that I have not grasped?
> 
>> The attach mechanism was of course never intended to be used this way.
>> It was meant for troubleshooting tools and profilers/similar to load
>> agents into running VMs. Back in the JDK 6 then we did consider
>> disallowing attaching to the current VM but didn't enforce it - one
>> reason is that it's not hard to just fork a VM with tools.jar on the
>> class path and connect back to the parent.
> 
> Well, I understand that this is not what you intended. However, i) it
> turns out to have been very useful that it does work this way and ii)
> stopping it doing so has a cost which needs to be taken into account --
> at the very least by giving those who have been relying on it for quite
> some time to manage their business concerns to adjust to the change.
> 
>> So that is the context for the discussion. We need to find a good way to
>> put the Genie back in its bottle. It may be that we have to disable
>> attaching to the current or ancestor VMs. We may have to prohibit the
>> instrumentation of core modules by late binding agents. We may have to
>> do some disabling of agent loading. Maybe a combination. Suggestions and
>> proposals are of course welcome.
> 
> I'm very happy to consider all sorts of half-way houses or even -- in
> time -- the full change recommended in the original JIRA. For example,
> rejecting instrumentation in some specific set of bootstrap/JDK module
> classes (like, say, java.base) from an agent which has been dynamically
> loaded might well be a workable compromise -- that at least allows users
> to employ an agent to tweak any code that is in the classpath through
> their choice.
> 
> However, whatever compromise we arrive at I'd certainly like time to
> adjust to it and hence prefer to see it happen in JDK10.
> 
> regards,
> 
> 
> Andrew Dinn
> -----------
> Senior Principal Software Engineer
> Red Hat UK Ltd
> Registered in England and Wales under Company Registration No. 03798903
> Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander