Eliminating the security overhead when not running with a security manager

[Alan Bateman]

One of the long standing issues with the security manager support is 
that the overhead when there is no security manager is non-zero. Two 
specific examples are (i) the overhead of defineClass (assuming the 
defining loader is a SecureClassLoader) as it involves a callback to get 
the permissions for the protection domain and (ii) the overhead of 
AccessController.doPrivileged in order to execute an action on a 
privileged stack.

The bigger question is of course whether it is interesting to run with a 
security manager in 2017. It's clearly still important for environments 
that download potentially malicious code to execute in a sandox but that 
isn't most applications these days. We have seen a few cases where 
applications set a security manager in order to enforce some policy, 
like preventing plugins calling System.exit but these are cases that 
would be better served with other solutions.

I would like to explore changes to the API and implementation that would 
allow us to eliminate some of the overhead when not running with a 
security manager. Long term then it would be interesting to explore 
degrading and eventually dropping the security manager but that is 
beyond the scope of what I would like to discuss here. Sean Mullan and 
Jeff Nisewanger ran an interesting BOF at JavaOne 2017 on this topic and 
I believe are planning to do a survey at some point to understand the 
current usage of the security manager.

For now I would like to explore what it would take to eliminate some of 
the overhead. A big challenge is the System.setSecurityManager API as 
allows a security manager to be set in a running VM. This means that 
classes loaded before a security manager is set may be involved in 
permission checks that happen after a security manager is set. 
Similarly, privileged actions may have started before the security 
manager is set. The complications around this go away if there was some 
way to know that a security manager can never be turned on in a running 
system.

To get started, I've put a prototype of an initial proposal on the cr 
site [1]. The summary of the proposal is:

1. A "disallow security manager" mode that is opt-in to disallow 
setSecurityManager being used to set a security manager in a running VM. 
Opt-in means it has not impact on existing code.

2. Deprecates setSecurityManager to discourage further usage and allow 
for the possibility of making "disallow security manager" the default in 
the future. If that were to become the default then it would have no 
impact on deployments that set the security manager on the command line 
at startup.

3. When running in "disallow security manager" mode, 
AccessController.doPrivileged invokes actions directly and 
SecureClassLoader.defineClass skips the callback to get permissions for 
the protection domain. This part helps startup and runtime performance.

It's important not to get hung up on the details in the webrev, the 
important thing is understand if the security folks on this mailing list 
are open to a run mode that prevents a security manager being set in a 
running system.

-Alan.

[1] http://cr.openjdk.java.net/~alanb/8191053/webrev/