Interoperation (Was: Re: Module-system requirements)

[David M. Lloyd]

On 03/16/2015 03:43 PM, mark.reinhold at oracle.com wrote:
> 2015/3/11 6:14 -0700, david.lloyd at redhat.com:
>> On 03/09/2015 03:54 PM, mark.reinhold at oracle.com wrote:
>>> 2015/2/16 5:57 -0800, david.lloyd at redhat.com:
>>>> ...
>>>>
>>>> In the case of both Java EE and OSGi, I think it's safe to say that you
>>>> need to apply a strict 1:1 mapping between modules and class loaders for
>>>> this to work.
>>>
>>> I don't think we can mandate this.  To divide the SE platform into
>>> modules in a compatible way requires the ability to load multiple modules
>>> via the same class loader, so as to respect long-standing assumptions
>>> about which platform classes are loaded by the built-in class loaders
>>> (i.e., the bootstrap and extension loaders).  In an early Jigsaw
>>> prototype we tried the 1:1 module/loader mapping for platform modules
>>> and found it unworkable from a compatibility perspective.
>>
>> I went through the jigsaw list archives looking for this discussion and
>> didn't turn up anything; was this discussed on-list and I am just
>> failing to find it?
>
> I don't recall it being discussed on-list.
>
> Perhaps the biggest issue we ran into is that there's lots of code, both
> in the JDK and out in the wild, that does tests of the form
>
>      if (ob.getClass().getClassLoader() == null) { ... }
>
> which, of course, succeeds for classes loaded by the bootstrap loader.
>
> We expect to divide the bootstrap classes into multiple modules (see JEP
> 200 [1] and the modules.xml file in JDK 9 [2] for the current prototype
> definition), which would break all such tests if each module had its own
> class loader.
>
> [1] http://openjdk.java.net/jeps/200
> [2] http://hg.openjdk.java.net/jdk9/jdk9/file/tip/modules.xml

So it sounds like there is a more concrete underlying requirement here:

- Define a set of "bootstrap modules" that must all return null for 
getClassLoader()
- Using ClassLoader.getSystemResource*(*) and Class.forName(*,*,null) 
must search all "bootstrap modules"
- The "java.base" module must be loaded from the bootstrap class loader 
for implementation reasons
- "Bootstrap modules" need not actually be loaded by the bootstrap CL if 
the above requirements are met

Side requirements:

- A mechanism is required to get the module for a given class (if module 
= class loader, this means non-null for bootstrap modules (maybe for 
java.base as well?))

The way this would play out with a module system like ours is, I'd still 
have 1:1 class loaders, but I'd have three module loaders at boot: the 
first only "loads" (wraps really) "java.base"; the second loads all the 
"bootstrap modules" which have the above special requirements and 
(uniquely) privileges to carry them out; and the third layer is the 
"regular" module loader which is used to load the remaining JDK modules 
(application distribution modules would probably be a layer above, I 
guess).  The module loader implementations would live in "java.base".

IOW I don't think that the above constraints alone actually mandate a 
relationship *or* a lack of relationship between modules and class loaders.

>>> Aside from the SE platform itself, I don't think we can force all
>>> applications to adopt the unique-class-loader-per-module approach.
>>> Oracle's WebLogic application server, e.g., doesn't do that today and
>>> it'd be difficult, if not impossible, to modify it to do so without
>>> breaking existing applications.
>>
>> I guess I can't really comment on what WebLogic does or does not do.  I
>> can say however that we were able to move from a flat classpath
>> architecture to a modular architecture like this without breaking things
>> due to the modularization.  The Java EE spec is pretty clear about what
>> needs to be visible to what, and by translating this information into
>> module dependency information, we were able to make a smooth transition.
>>    I can't really imagine an application that would be broken unless they
>> were relying on some specific non-spec behavior as the EE spec says:
>>
>> "[...] A Java EE application should not assume that all components in
>> the application will be available on the class path of the application
>> at run time. Each component might be loaded into a separate class loader
>> with a separate namespace. [...]"
>
> The concern with WebLogic is that existing customers rely upon aspects of
> its particular class-loader hierarchy, beyond what is guaranteed by the
> Java EE specification.  Changing it incompatibly would break those
> customers' applications.
>
>> Note that in the EE spec, the term "component" has a finer granularity
>> than "module", i.e. there are 1..N components per module, whereas an
>> "application" can contain 1 or more modules.
>
> Yes, and I don't think that the module concept we're discussing here will
> wind up corresponding exactly to what EE calls "modules".

The two above statements, taken together, don't really exclude class 
loader per module.  Think of it instead as "module per class loader" - 
any place you use a class loader today, you could instead use a module. 
  The parent relationship of a module class loader could easily be set 
up to reflect any implementation realities that must be dealt with.

However if "modular EE" is indeed a completely separate mode of 
operation then it seems like WebLogic (or any other vendor) would not 
have anything to worry about.  We'd still be free to use SE modules for 
EE modules in "legacy mode", whereas other containers would be free to 
continue as they have been.

>
>> ...
>>
>>> I suggest the module system be as neutral as possible with respect to how
>>> modules and class loaders are related.  A complex application should be
>>> able to spin up a class loader for every module it loads, if needed
>>> (excepting non-upgradeable platform classes), but not all applications
>>> should be forced to do so.
>>>
>>> New requirement, for the Dynamic Configuration section:
>>>
>>> - _Control of class loaders_ --- An application that loads a module
>>> into a dynamic configuration must be able to specify the class loader
>>> to be used to load that module.  That loader need not be the same as
>>> the loader used to load any other module in the configuration.
>>
>> This goes part of the way, for sure.  I'm not 100% sold yet though.
>> It's not easy to let go of something that (from our perspective) is
>> tried and true.
>
> This requirement allows you to use the 1:1 model above SE itself, if you
> want that, but it doesn't force everyone else to do so.

I just wonder how much difference there really is between composing 
single modules out of multiple collections of classes and resources (as 
we do), versus composing single class loaders out of modules (as you 
propose).  Given the *high* amount of conceptual overlap between modules 
and class loaders, I'm still inclined to reject the idea that we need 
another layer there based on the requirements put forth.  While giving 
the option to define the association sounds good, the fact that the 
option even *exists* means that everything using class loaders to load 
classes and resources (read: everything, period) is now broken, but I 
don't feel like there's a good justification for this that isn't either 
self-referential or a broadening of the actual minimum requirements (see 
above).

>>> ...
>>>
>>>> OSGi specifically requires that bundle class loaders implement an OSGi
>>>> spec interface, implying that the module class loader class must be
>>>> extensible (as I said a couple of years back though, we make most of our
>>>> methods final on the module class loader to disallow any really bizarre
>>>> behavior, and this works well in practice).
>>>
>>> Which OSGi interface are you thinking of here?
>>
>> In OSGi Core 6.0.0 section 3.9.9:
>>
>> "Class Loader - An OSGi framework must ensure that the class loader of a
>> class that comes from
>> a bundle implements the BundleReference interface."
>>
>> The BundleReference interface has a single getBundle() method on it.
>
> Ah, got it.
>
>>>> We also had to implement a
>>>> pre- and/or post-class-define hook per module to allow some OSGi magic
>>>> to happen.
>>>
>>> These are non-final protected methods in your module class loader?
>>>
>>> Could you forward a description of these methods to the list?
>>
>> Yeah, they look like this:
>>
>>       protected void preDefine(ClassSpec classSpec, String className) {
>>       }
>>
>>       protected void postDefine(ClassSpec classSpec, Class<?> definedClass) {
>>       }
>>
>> ClassSpec contains the bytes and CodeSource of the class.  The hooks
>> don't have opportunity to do anything like rewrite the class bytes, but
>> we also accept a java.lang.instrument.ClassFileTransformer instance in
>> the module definition for that purpose (though we aren't using it at
>> this point).  Our OSGi implementation only ended up using one of these
>> hooks (preDefine I think).  They were added to support lazy bundle
>> activation.
>
> Okay.
>
>>>> Finally there is a per-module fallback resource loader that
>>>> our OSGi implementation used for one particular part of the spec.
>>>
>>> Which part, in particular?
>>
>> I'll have to go back through my notes to say for sure, but I think we
>> used it for a couple of purposes:
>>
>> 1) To support dynamic imports: https://issues.jboss.org/browse/MODULES-30
>> 2) As a general hook to allow for OSGi-style "lazy" packages (the loader
>> would examine the package of the class in question, recalculate the
>> package set, and then retry the class load):
>> https://issues.jboss.org/browse/MODULES-63
>
> Interesting.
>
>>> To what extent does OSGi assume that bundles "export" their resources?
>>> If we decide that a module's resources are not available outside the
>>> module itself then will the approach of extending a module class loader
>>> to be an OSGi bundle loader even work?
>>
>> This I will defer to an OSGi expert.  My understanding is that OSGi
>> largely operates on a package basis only, but I don't know how resources
>> within the packages fit in to this picture (if at all).
>
> §3.9.6 and §10.1.5.13 of the OSGi R6 specification imply that all the
> resources in a bundle are available to anyone with access to the bundle
> object itself and, if running with a security manager, the appropriate
> OSGi AdminPermission(s).
>
> If we adopt the "Access to class files and resources" requirement which I
> suggested in the "exporting things" thread then an OSGi implementation
> built on top of the platform module system could use that to expose the
> resources of a module artifact as needed.
>
>>>> Java EE's requirements are much simpler and are satisfied by
>>>> module-granularity dependencies and (in certain cases) resource importing.
>>>
>>> Aside from things like deployment descriptors, which are interpreted by
>>> Java EE implementations, does the EE specification require EE "modules"
>>> to export resources to other modules?
>>
>> The Java EE 7 platform specification does seem to do so, with language
>> such as "Components in the web container may have access to the
>> following classes and resources. [...] The classes and resources
>> accessible to any other web modules included in the
>> same ear file, as described above."
>>
>> The spec also seems to use the term "contents" interchangeably with
>> "classes and resources".
>
> The Java EE specification, just like OSGi, is explicitly built on the JAR
> file format and derivations thereof, so I think the same solution applies
> for Java EE: An EE implementation can access a module's artifact and
> expose the artifact's resources as needed.
>
> - Mark
>

-- 
- DML