Further Defenses for the Security Manager

[Michael Maass]

Hello All,

I've spent the last 6 months working with some colleagues on a project 
that aimed to stop an exploitation avenue that has been popular with 
recent Java exploits: disabling the security manager. We think that what 
ended up with may be worthy of a JEP and/or a prototype implementation 
in the JVM; we're looking for feedback from the experts, you folks.

We initially set out to determine how benign applications interact with 
the security manager and to determine how those interactions differ from 
malicious ones. The idea was that an investigation aimed at making these 
determinations could show us ways to constrain operations on the manager 
that are backwards compatible with benign applications while blocking 
exploits.

After investigating the open source Java landscape we learned to 
distinguish between (1) "self-protecting" and "defenseless" security 
managers and (2) the operations that may be performed in the presence of 
each. A defenseless manager enforces a policy that uses at least one 
permission in a set of permissions we identified that allow anyone to 
change the manager itself or the policy it enforces (a self-protecting 
manager is the opposite of a defenseless manager). Given this 
definition, we found that enforcing the following rules stopped known 
exploits without impeding (aside from overhead) the execution of benign 
applications:

1. If a self-protecting manager is set, a class may not load or cause 
the loading of a class more privileged than itself unless the loaded 
class is in a protected package (listed in the package.access property 
in java.security.Security).

2. If a self-protecting manager is set, an application may not change 
the security manager or the enforced policy in any way.

We built a JVMTI agent to enforce these rules (available here: 
https://github.com/SecurityManagerCodeBase/JavaSandboxFortifier). 
Unfortunately, to use the JVMTI events that allow us to enforce rule 2, 
the JIT must be off, which drastically slows down the execution of 
applets in the presence of our agent. Rule 1 adds about 1-2% overhead. 
We believe overhead would drop enough for adoption if the JVM were to 
natively enforce these rules instead of using an agent, but we do not 
have the resources or the expertise to try this ourselves.

Any thoughts on whether or not this type of change is worth pursing in 
the form of a JEP and/or prototype in the JVM? If so, is anyone willing 
and able to help?

Thanks,

Michael Maass, PhD Candidate
Software Engineering
Institute for Software Research
School of Computer Science
Carnegie Mellon University