Further Defenses for the Security Manager

Michael Maass mmaass at andrew.cmu.edu
Thu Jan 8 21:29:28 UTC 2015

Hello Jeff,

Sorry for the delayed response.

Regarding privilege escalation and class loading, our observation was 
that many Java exploits have an exploit class and a separate payload 
class. The exploit class typically attacks a vulnerability with the 
outcome that the payload class is loaded with elevated privileges (e.g. 
use a vulnerability to create a privileged ClassLoader, then use the 
privileged loader to load the payload). The exploit class then typically 
executes the payload as a PrivilegedAction.

The payload at this point is running with AllPermissions. This 
privileged position is typically leveraged to disable the sandbox before 
moving on to different stages that will download and install more 
advanced malware. However, there is no obligation at this point to 
disable the sandbox in the payload, the payload class could simply 
perform the malicious operations straight-away as the class is 
effectively not sandboxed anyway (i.e. it's not sufficient to simply 
stop applets from setting the SecurityManager to null).

Our thought was to catch cases where a payload class, which should have 
the same privileges as the rest of the applet's classes, is loaded with 
a superset of those privileges. Given this goal, I am not entirely sure 
these points apply as potential issues. For example, if the exploit 
class were preloaded by the JRE, wouldn't it be preloaded with the same 
privileges as the rest of the applet, thus not granting it the ability 
to perform its malicious operations? If I understand correctly, the same 
would be true for cases 1-4.

Our specific and perhaps naive idea was that the JVM could inspect the 
stack frames when a class is being loaded to ensure that the privileges 
of the loaded class are a subset of the privileges for each class in the 
stack frames. We did not believe this needed to account for doPrivileged 
because it is our understanding that a class that can use a ClassLoader 
to load another class with more privileges and directly access the 
loaded, privileged class will always break the Java security model.

Does this paint a reasonable picture? My understanding of JVM internals 
is unfortunately not very advanced.



On 12/19/2014 07:47 PM, Jeff Nisewanger wrote:
> Thank you for contacting us and sharing the initial results of your research.
> You raised two basic topics.
> First, you discussed the possibility of adding additional restrictions on the ability to change the system Security Manager multiple times during application execution. This is normally already subject to a security check but there might be malicious exploits that bypass the existing check.
> It turns out that we are already independently investigating this topic. We agree that such a restriction likely requires additional assistance from the JVM implementation for several reasons.
> As you mentioned, an important aspect to such a change is to minimize backwards compatibility problems. Your research on the use of security managers by existing Java applications is interesting and useful.
> You also suggested the general idea that whenever a new class is loaded we should check the granted permissions of the classes containing the methods on the thread call stack.
> You propose to ensure that each such class on the call stack has already itself been granted a superset of the permissions that would be granted to the newly loaded class. However, you would not enforce this general rule for common cases such as classes loaded by the bootstrap loader or classes within the list of restricted package names.
> Does your proposal take into account any calls to AccessController.doPrivileged() on the thread call stack? Could a thread shortcut this proposed check by causing it's action to take place on another thread (create a new thread to initiate the class loading)?
> The Java Virtual Machine specification allows for class loading to be initiated in a flexible manner that may have only a tangential relationship to the members of a thread's call stack. Examples include:
> 1. When a class is being loaded the JVM resolves the names of the superclass or super interfaces
> 2. Constant pool resolution during the execution of a bytecode instruction in a class that refers symbolically to a named class type.
> 3. Bytecode verification of another class which is in the process of being loaded. To ensure type-safety, the verifier may sometimes load other classes recursively that are symbolically referenced by a bytecode instruction (before any of the code in the class being verified is ever executed normally).
> 4. Direct calls to Class.forName(), ClassLoader.loadClass() or perhaps somehow maliciously to ClassLoader.defineClass().
> Also, according to the specification, the JVM is free to spontaneously "pre-load" classes whenever it feels like it even if those classes have no apparent relationship with the thread chosen to load them.
> For this reason, Java class loading is generally independent of thread context except for the immediate class in which a constant pool resolution is being performed which is subject, for example, to the language-level access controls (public, protected, etc.) of the class it is referencing.
> Thanks,
> Jeff
>> On Oct 29, 2014, at 6:35 AM, Michael Maass <mmaass at andrew.cmu.edu> wrote:
>> Hello All,
>> I've spent the last 6 months working with some colleagues on a project that aimed to stop an exploitation avenue that has been popular with recent Java exploits: disabling the security manager. We think that what ended up with may be worthy of a JEP and/or a prototype implementation in the JVM; we're looking for feedback from the experts, you folks.
>> We initially set out to determine how benign applications interact with the security manager and to determine how those interactions differ from malicious ones. The idea was that an investigation aimed at making these determinations could show us ways to constrain operations on the manager that are backwards compatible with benign applications while blocking exploits.
>> After investigating the open source Java landscape we learned to distinguish between (1) "self-protecting" and "defenseless" security managers and (2) the operations that may be performed in the presence of each. A defenseless manager enforces a policy that uses at least one permission in a set of permissions we identified that allow anyone to change the manager itself or the policy it enforces (a self-protecting manager is the opposite of a defenseless manager). Given this definition, we found that enforcing the following rules stopped known exploits without impeding (aside from overhead) the execution of benign applications:
>> 1. If a self-protecting manager is set, a class may not load or cause the loading of a class more privileged than itself unless the loaded class is in a protected package (listed in the package.access property in java.security.Security).
>> 2. If a self-protecting manager is set, an application may not change the security manager or the enforced policy in any way.
>> We built a JVMTI agent to enforce these rules (available here: https://github.com/SecurityManagerCodeBase/JavaSandboxFortifier). Unfortunately, to use the JVMTI events that allow us to enforce rule 2, the JIT must be off, which drastically slows down the execution of applets in the presence of our agent. Rule 1 adds about 1-2% overhead. We believe overhead would drop enough for adoption if the JVM were to natively enforce these rules instead of using an agent, but we do not have the resources or the expertise to try this ourselves.
>> Any thoughts on whether or not this type of change is worth pursing in the form of a JEP and/or prototype in the JVM? If so, is anyone willing and able to help?
>> Thanks,
>> Michael Maass, PhD Candidate
>> Software Engineering
>> Institute for Software Research
>> School of Computer Science
>> Carnegie Mellon University

More information about the security-dev mailing list