Coupling modules and class loaders

[David M. Lloyd]

On 12/17/2015 10:26 AM, mark.reinhold at oracle.com wrote:
> 2015/12/15 9:50 -0800, david.lloyd at redhat.com:
>> On 12/14/2015 01:44 PM, mark.reinhold at oracle.com wrote:
>>> ...
>>>
>>> What I'm thinking of is an existing application with its own non-trivial
>>> class-loader architecture which, due to compatibility constraints, cannot
>>> be changed.  If at least one of its class loaders loads classes from
>>> logically-distinct components that are later converted into corresponding
>>> modules then upgrading the application to load those components as
>>> modules would, under your approach, require replacing that one class
>>> loader with many class loaders, which would break compatibility.
>>>
>>> Imagine, e.g., a Java EE application server that loads implementations of
>>> all the standard EE components using one class loader.  Imagine further
>>> that existing users (for good or for ill) depend upon that.  If a future
>>> version of Java EE defines those components as modules then the only way
>>> to arrange for this app server to load them as such would be to change
>>> its class-loader architecture in a way that would break existing uses.
>>>
>>> (This is, roughly, the situation with Oracle's WebLogic app server, as
>>> I've mentioned previously.)
>>
>> With a multi-class-loader arrangement, it should be possible to emulate
>> any single-class-loader arrangement, in two ways and with one caveat
>> that I know of:
>>
>> * You can create modules which provide "cut down" views of another
>> module, such that the module appears as multiple modules but with one
>> /defining/ class loader for all classes actually loaded from them;
>
> I think you mean one defining loader, not one initiating loader.  If
> loader M actually loads a module, and loaders X and Y define "cut down"
> views of M by delegating to M, then all three loaders can be initiating
> loaders of classes in M but only M will be the defining loader of such
> classes.

You're right, I used the wrong term, I don't know what I was thinking 
about.  Sorry about that, I've fixed the text to hopefully reduce 
confusion...

>> * You can create a module which aggregates multiple modules into one, as
>> long as no code relies on the the specific relationship between the
>> /defining/ class loaders of classes within the module (as the /defining/
>> class loader will reflect the module in which the class was defined;
>> this is the caveat I was referring to above).
>
> Again, I think you mean defining loader.  If loaders M1 and M2 load
> modules, and loader X aggregates those into a single apparent module,
> then M1 and M2 will be defining loaders and X will be an initiating
> loader.

Fixed...

>
>> In the JDK itself, there's the special case of the null return of
>> Class.getClassLoader() being used for legacy security checks, for code
>> that depends on this, though there are multiple ways that I know of in
>> which this could potentially be mitigated, if this is a problem that you
>> are thinking of and you want to discuss it.
>>
>> What do you think about these options in terms of WebLogic?  Are there
>> additional cases that cannot be covered by fixes in these two areas?
>> Are there other compatibility cases beyond these that you have in mind
>> where using multiple class loaders for either platform components or for
>> Java EE specification modules could break existing behaviors, or does
>> the combination of the change of /defining/ class loader and behavior of
>> Class.getClassLoader() summarize the potential issues that you see?
>
> To make an approach along these lines work for the JDK, for WebLogic,
> and for any other application with a non-trivial, non-loader-per-module
> class-loader architecture, I think you're going to have to fundamentally
> redefine the meaning of the Class::getClassLoader method.

...because WebLogic depends on the defining class loader being something 
in particular?  Looking at the WebLogic class loading documentation, it 
seems like it is less important what the defining loader is, and more 
important what each (EE) module can "see".  In fact there are a number 
of restrictions which actually seem to play in favor of per-classloader 
modules: limited nesting depth, a clear delegation pattern (because 
traditional classloader hierarchy trees are just a subclass of the 
overall directed-graph relationship of modularity), restriction of 
customization to web and EJB modules, call-by-value in certain 
situations, etc.

It would appear to me (just from reading the documentation) that 
WebLogic should already be well-positioned to adopt a 
classloader-oriented modularity strategy for Java EE, unless there is 
some problem that stems directly from having differing defining class 
loaders.

> It would no
> longer return the defining loader of a class but, rather, an initiating,
> aggregating loader as you describe above. Given the roles of class
> loaders in both VM-level access checking and SecurityManager-level
> permission checking, this is far from being a simple change.
>
> To take just one case, suppose a class C is loaded and defined by a
> module class loader M1, which itself is aggregated together with some
> other module class loader M2 by an aggregating loader X, so if I have
> C's Class object then its getClassLoader method will return X.  Suppose
> further that X is an instance of an end-user-supplied subclass of a
> ClassLoader subclass provided by the application, long ago, as part of
> its supported API.  When one of the defineClass methods of X is invoked
> to define some new class D, what will be the defining loader of D?
>
> If the defining loader is not X, then X must somehow delegate class
> definition to either M1 or M2, but which one?  If D's package is already
> defined to just one of those loaders then maybe defining it with that
> loader would (often) be the right thing.  What would X do if D's package
> is split across M1 and M2, or would you disallow that?  What would X do
> if D's package is not yet defined in either M1 or M2, just define D with
> X?  That might work in some cases but not if some code later on depends
> upon D having been defined by M1 specifically.
>
> If the defining loader is X, rather than M1 or M2, then D would have a
> different defining loader relative to classes now in M1 or M2 than it did
> when run on pre-9 releases.  If D is meant to have package-private access
> to some class E in the same package, whose defining loader was X but is
> now M1 due to modularization, then it will fail, since D's run-time
> package will be different from that of E, since its defining loader is
> different.
>
> (You could consider an even deeper change and say that X is the defining
>   loader, not just an initiating loader, of all classes in both M1 and M2.
>   Module-private types in those modules would then, however, no longer be
>   strongly encapsulated.)
>
> So, that's just one case of one family of methods, defineClass, in the
> existing ClassLoader API.  To change the meaning of Class::getClassLoader
> would require a deep analysis of all the methods in ClassLoader, not just
> this one.  Maybe there's some super-clever way to make them all work in a
> manner that's both sensible and compatible, but I don't know what that is
> and I can't believe it'd be straightforward.  This kind of complexity is
> exactly why we abandoned the module-per-loader approach implemented in
> the first Jigsaw prototype.

Yeah I didn't mean to imply that we should make a fundamental shift in 
what getClassLoader means; however, I would definitely consider the idea 
of making small changes to it to be reasonable, if it were just for 
compatibility with things that use e.g. ClassLoader.getSystemResources() 
or Class.forName(x, y, null) for example, or which use getClassLoader() 
== null for security checks; this kind of behavior could optionally be 
deprecated & removed in the long term.

The thrust of my question was intended to be regarding platform and 
application class loaders.  As a hopefully better approach, I'll break 
it into a few (more specific) hypothetical situations which are centered 
around 1:1 modules/classloaders.  Some seem reasonable, some seem more 
unlikely but might still be worth exploring for the purposes of thought 
experimentation.

1. Premise: The Java EE 9 specification establishes a rule wherein each 
Java EE API is required to be available statically to applications by a 
module name which corresponds to the API in use.  Existing application 
servers currently present all these APIs as one large unit.  Question: 
is it a problem to present each API as a "cut down"/filtered view of one 
module which consists the of all those APIs in the same large unit?

2. Premise: Same as #1, except the Java EE 9 specification also requires 
that each API be actually encapsulated in a separate module.  Question: 
is it a problem that the defining class loader of these classes has 
changed to be per-API rather than a single one?

3. Premise: Java EE 9 requires that each deployment item presently 
defined in the EE platform spec as a "module" be established as a 
module, while otherwise maintaining the existing neutrality about the 
"types and arrangements" of class loaders (meaning for example that it 
is the app vendor's choice as to whether each transitive Class-Path 
dependency becomes a (potentially shared) module, or whether those 
classes are folded in to the EE module, or some other strategy). 
Question: does this present a compatibility problem that you can foresee?

4. Premise: Same as #3, however it also introduces a means to determine 
whether other JARs in a deployment should be established as a module, as 
well as a means to establish dependencies to and from such modules. 
Question: does this mitigate or exacerbate any problems from #3?

5. Premise: Java EE 9 requires that every single archive within every 
application or module be established as a module.  Question: how big a 
disaster would this be, and why?

-- 
- DML