Coupling modules and class loaders

[David M. Lloyd]

Still awaiting a reply on the following.  I still maintain that the 
module API and class loader decoupling is a design issue.

On 12/17/2015 12:20 PM, David M. Lloyd wrote:
> On 12/17/2015 10:26 AM, mark.reinhold at oracle.com wrote:
>> 2015/12/15 9:50 -0800, david.lloyd at redhat.com:
>>> On 12/14/2015 01:44 PM, mark.reinhold at oracle.com wrote:
>>>> ...
>>>>
>>>> What I'm thinking of is an existing application with its own
>>>> non-trivial
>>>> class-loader architecture which, due to compatibility constraints,
>>>> cannot
>>>> be changed.  If at least one of its class loaders loads classes from
>>>> logically-distinct components that are later converted into
>>>> corresponding
>>>> modules then upgrading the application to load those components as
>>>> modules would, under your approach, require replacing that one class
>>>> loader with many class loaders, which would break compatibility.
>>>>
>>>> Imagine, e.g., a Java EE application server that loads
>>>> implementations of
>>>> all the standard EE components using one class loader.  Imagine further
>>>> that existing users (for good or for ill) depend upon that.  If a
>>>> future
>>>> version of Java EE defines those components as modules then the only
>>>> way
>>>> to arrange for this app server to load them as such would be to change
>>>> its class-loader architecture in a way that would break existing uses.
>>>>
>>>> (This is, roughly, the situation with Oracle's WebLogic app server, as
>>>> I've mentioned previously.)
>>>
>>> With a multi-class-loader arrangement, it should be possible to emulate
>>> any single-class-loader arrangement, in two ways and with one caveat
>>> that I know of:
>>>
>>> * You can create modules which provide "cut down" views of another
>>> module, such that the module appears as multiple modules but with one
>>> /defining/ class loader for all classes actually loaded from them;
>>
>> I think you mean one defining loader, not one initiating loader.  If
>> loader M actually loads a module, and loaders X and Y define "cut down"
>> views of M by delegating to M, then all three loaders can be initiating
>> loaders of classes in M but only M will be the defining loader of such
>> classes.
>
> You're right, I used the wrong term, I don't know what I was thinking
> about.  Sorry about that, I've fixed the text to hopefully reduce
> confusion...
>
>>> * You can create a module which aggregates multiple modules into one, as
>>> long as no code relies on the the specific relationship between the
>>> /defining/ class loaders of classes within the module (as the /defining/
>>> class loader will reflect the module in which the class was defined;
>>> this is the caveat I was referring to above).
>>
>> Again, I think you mean defining loader.  If loaders M1 and M2 load
>> modules, and loader X aggregates those into a single apparent module,
>> then M1 and M2 will be defining loaders and X will be an initiating
>> loader.
>
> Fixed...
>
>>
>>> In the JDK itself, there's the special case of the null return of
>>> Class.getClassLoader() being used for legacy security checks, for code
>>> that depends on this, though there are multiple ways that I know of in
>>> which this could potentially be mitigated, if this is a problem that you
>>> are thinking of and you want to discuss it.
>>>
>>> What do you think about these options in terms of WebLogic?  Are there
>>> additional cases that cannot be covered by fixes in these two areas?
>>> Are there other compatibility cases beyond these that you have in mind
>>> where using multiple class loaders for either platform components or for
>>> Java EE specification modules could break existing behaviors, or does
>>> the combination of the change of /defining/ class loader and behavior of
>>> Class.getClassLoader() summarize the potential issues that you see?
>>
>> To make an approach along these lines work for the JDK, for WebLogic,
>> and for any other application with a non-trivial, non-loader-per-module
>> class-loader architecture, I think you're going to have to fundamentally
>> redefine the meaning of the Class::getClassLoader method.
>
> ...because WebLogic depends on the defining class loader being something
> in particular?  Looking at the WebLogic class loading documentation, it
> seems like it is less important what the defining loader is, and more
> important what each (EE) module can "see".  In fact there are a number
> of restrictions which actually seem to play in favor of per-classloader
> modules: limited nesting depth, a clear delegation pattern (because
> traditional classloader hierarchy trees are just a subclass of the
> overall directed-graph relationship of modularity), restriction of
> customization to web and EJB modules, call-by-value in certain
> situations, etc.
>
> It would appear to me (just from reading the documentation) that
> WebLogic should already be well-positioned to adopt a
> classloader-oriented modularity strategy for Java EE, unless there is
> some problem that stems directly from having differing defining class
> loaders.
>
>> It would no
>> longer return the defining loader of a class but, rather, an initiating,
>> aggregating loader as you describe above. Given the roles of class
>> loaders in both VM-level access checking and SecurityManager-level
>> permission checking, this is far from being a simple change.
>>
>> To take just one case, suppose a class C is loaded and defined by a
>> module class loader M1, which itself is aggregated together with some
>> other module class loader M2 by an aggregating loader X, so if I have
>> C's Class object then its getClassLoader method will return X.  Suppose
>> further that X is an instance of an end-user-supplied subclass of a
>> ClassLoader subclass provided by the application, long ago, as part of
>> its supported API.  When one of the defineClass methods of X is invoked
>> to define some new class D, what will be the defining loader of D?
>>
>> If the defining loader is not X, then X must somehow delegate class
>> definition to either M1 or M2, but which one?  If D's package is already
>> defined to just one of those loaders then maybe defining it with that
>> loader would (often) be the right thing.  What would X do if D's package
>> is split across M1 and M2, or would you disallow that?  What would X do
>> if D's package is not yet defined in either M1 or M2, just define D with
>> X?  That might work in some cases but not if some code later on depends
>> upon D having been defined by M1 specifically.
>>
>> If the defining loader is X, rather than M1 or M2, then D would have a
>> different defining loader relative to classes now in M1 or M2 than it did
>> when run on pre-9 releases.  If D is meant to have package-private access
>> to some class E in the same package, whose defining loader was X but is
>> now M1 due to modularization, then it will fail, since D's run-time
>> package will be different from that of E, since its defining loader is
>> different.
>>
>> (You could consider an even deeper change and say that X is the defining
>>   loader, not just an initiating loader, of all classes in both M1 and
>> M2.
>>   Module-private types in those modules would then, however, no longer be
>>   strongly encapsulated.)
>>
>> So, that's just one case of one family of methods, defineClass, in the
>> existing ClassLoader API.  To change the meaning of Class::getClassLoader
>> would require a deep analysis of all the methods in ClassLoader, not just
>> this one.  Maybe there's some super-clever way to make them all work in a
>> manner that's both sensible and compatible, but I don't know what that is
>> and I can't believe it'd be straightforward.  This kind of complexity is
>> exactly why we abandoned the module-per-loader approach implemented in
>> the first Jigsaw prototype.
>
> Yeah I didn't mean to imply that we should make a fundamental shift in
> what getClassLoader means; however, I would definitely consider the idea
> of making small changes to it to be reasonable, if it were just for
> compatibility with things that use e.g. ClassLoader.getSystemResources()
> or Class.forName(x, y, null) for example, or which use getClassLoader()
> == null for security checks; this kind of behavior could optionally be
> deprecated & removed in the long term.
>
> The thrust of my question was intended to be regarding platform and
> application class loaders.  As a hopefully better approach, I'll break
> it into a few (more specific) hypothetical situations which are centered
> around 1:1 modules/classloaders.  Some seem reasonable, some seem more
> unlikely but might still be worth exploring for the purposes of thought
> experimentation.
>
> 1. Premise: The Java EE 9 specification establishes a rule wherein each
> Java EE API is required to be available statically to applications by a
> module name which corresponds to the API in use.  Existing application
> servers currently present all these APIs as one large unit.  Question:
> is it a problem to present each API as a "cut down"/filtered view of one
> module which consists the of all those APIs in the same large unit?
>
> 2. Premise: Same as #1, except the Java EE 9 specification also requires
> that each API be actually encapsulated in a separate module.  Question:
> is it a problem that the defining class loader of these classes has
> changed to be per-API rather than a single one?
>
> 3. Premise: Java EE 9 requires that each deployment item presently
> defined in the EE platform spec as a "module" be established as a
> module, while otherwise maintaining the existing neutrality about the
> "types and arrangements" of class loaders (meaning for example that it
> is the app vendor's choice as to whether each transitive Class-Path
> dependency becomes a (potentially shared) module, or whether those
> classes are folded in to the EE module, or some other strategy).
> Question: does this present a compatibility problem that you can foresee?
>
> 4. Premise: Same as #3, however it also introduces a means to determine
> whether other JARs in a deployment should be established as a module, as
> well as a means to establish dependencies to and from such modules.
> Question: does this mitigate or exacerbate any problems from #3?
>
> 5. Premise: Java EE 9 requires that every single archive within every
> application or module be established as a module.  Question: how big a
> disaster would this be, and why?
>

-- 
- DML