Interoperation (Was: Re: Module-system requirements)

[mark.reinhold at oracle.com]

2015/3/11 6:14 -0700, david.lloyd at redhat.com:
> On 03/09/2015 03:54 PM, mark.reinhold at oracle.com wrote:
>> 2015/2/16 5:57 -0800, david.lloyd at redhat.com:
>>> ...
>>> 
>>> In the case of both Java EE and OSGi, I think it's safe to say that you
>>> need to apply a strict 1:1 mapping between modules and class loaders for
>>> this to work.
>> 
>> I don't think we can mandate this.  To divide the SE platform into
>> modules in a compatible way requires the ability to load multiple modules
>> via the same class loader, so as to respect long-standing assumptions
>> about which platform classes are loaded by the built-in class loaders
>> (i.e., the bootstrap and extension loaders).  In an early Jigsaw
>> prototype we tried the 1:1 module/loader mapping for platform modules
>> and found it unworkable from a compatibility perspective.
> 
> I went through the jigsaw list archives looking for this discussion and 
> didn't turn up anything; was this discussed on-list and I am just 
> failing to find it?

I don't recall it being discussed on-list.

Perhaps the biggest issue we ran into is that there's lots of code, both
in the JDK and out in the wild, that does tests of the form

    if (ob.getClass().getClassLoader() == null) { ... }

which, of course, succeeds for classes loaded by the bootstrap loader.

We expect to divide the bootstrap classes into multiple modules (see JEP
200 [1] and the modules.xml file in JDK 9 [2] for the current prototype
definition), which would break all such tests if each module had its own
class loader.

[1] http://openjdk.java.net/jeps/200
[2] http://hg.openjdk.java.net/jdk9/jdk9/file/tip/modules.xml

>> Aside from the SE platform itself, I don't think we can force all
>> applications to adopt the unique-class-loader-per-module approach.
>> Oracle's WebLogic application server, e.g., doesn't do that today and
>> it'd be difficult, if not impossible, to modify it to do so without
>> breaking existing applications.
> 
> I guess I can't really comment on what WebLogic does or does not do.  I 
> can say however that we were able to move from a flat classpath 
> architecture to a modular architecture like this without breaking things 
> due to the modularization.  The Java EE spec is pretty clear about what 
> needs to be visible to what, and by translating this information into 
> module dependency information, we were able to make a smooth transition. 
>   I can't really imagine an application that would be broken unless they 
> were relying on some specific non-spec behavior as the EE spec says:
> 
> "[...] A Java EE application should not assume that all components in 
> the application will be available on the class path of the application 
> at run time. Each component might be loaded into a separate class loader 
> with a separate namespace. [...]"

The concern with WebLogic is that existing customers rely upon aspects of
its particular class-loader hierarchy, beyond what is guaranteed by the
Java EE specification.  Changing it incompatibly would break those
customers' applications.

> Note that in the EE spec, the term "component" has a finer granularity 
> than "module", i.e. there are 1..N components per module, whereas an 
> "application" can contain 1 or more modules.

Yes, and I don't think that the module concept we're discussing here will
wind up corresponding exactly to what EE calls "modules".

> ...
> 
>> I suggest the module system be as neutral as possible with respect to how
>> modules and class loaders are related.  A complex application should be
>> able to spin up a class loader for every module it loads, if needed
>> (excepting non-upgradeable platform classes), but not all applications
>> should be forced to do so.
>> 
>> New requirement, for the Dynamic Configuration section:
>> 
>> - _Control of class loaders_ --- An application that loads a module
>> into a dynamic configuration must be able to specify the class loader
>> to be used to load that module.  That loader need not be the same as
>> the loader used to load any other module in the configuration.
> 
> This goes part of the way, for sure.  I'm not 100% sold yet though. 
> It's not easy to let go of something that (from our perspective) is 
> tried and true.

This requirement allows you to use the 1:1 model above SE itself, if you
want that, but it doesn't force everyone else to do so.

>> ...
>> 
>>> OSGi specifically requires that bundle class loaders implement an OSGi
>>> spec interface, implying that the module class loader class must be
>>> extensible (as I said a couple of years back though, we make most of our
>>> methods final on the module class loader to disallow any really bizarre
>>> behavior, and this works well in practice).
>> 
>> Which OSGi interface are you thinking of here?
> 
> In OSGi Core 6.0.0 section 3.9.9:
> 
> "Class Loader - An OSGi framework must ensure that the class loader of a 
> class that comes from
> a bundle implements the BundleReference interface."
> 
> The BundleReference interface has a single getBundle() method on it.

Ah, got it.

>>> We also had to implement a
>>> pre- and/or post-class-define hook per module to allow some OSGi magic
>>> to happen.
>> 
>> These are non-final protected methods in your module class loader?
>> 
>> Could you forward a description of these methods to the list?
> 
> Yeah, they look like this:
> 
>      protected void preDefine(ClassSpec classSpec, String className) {
>      }
> 
>      protected void postDefine(ClassSpec classSpec, Class<?> definedClass) {
>      }
> 
> ClassSpec contains the bytes and CodeSource of the class.  The hooks 
> don't have opportunity to do anything like rewrite the class bytes, but 
> we also accept a java.lang.instrument.ClassFileTransformer instance in 
> the module definition for that purpose (though we aren't using it at 
> this point).  Our OSGi implementation only ended up using one of these 
> hooks (preDefine I think).  They were added to support lazy bundle 
> activation.

Okay.

>>> Finally there is a per-module fallback resource loader that
>>> our OSGi implementation used for one particular part of the spec.
>> 
>> Which part, in particular?
> 
> I'll have to go back through my notes to say for sure, but I think we 
> used it for a couple of purposes:
> 
> 1) To support dynamic imports: https://issues.jboss.org/browse/MODULES-30
> 2) As a general hook to allow for OSGi-style "lazy" packages (the loader 
> would examine the package of the class in question, recalculate the 
> package set, and then retry the class load): 
> https://issues.jboss.org/browse/MODULES-63

Interesting.

>> To what extent does OSGi assume that bundles "export" their resources?
>> If we decide that a module's resources are not available outside the
>> module itself then will the approach of extending a module class loader
>> to be an OSGi bundle loader even work?
> 
> This I will defer to an OSGi expert.  My understanding is that OSGi 
> largely operates on a package basis only, but I don't know how resources 
> within the packages fit in to this picture (if at all).

§3.9.6 and §10.1.5.13 of the OSGi R6 specification imply that all the
resources in a bundle are available to anyone with access to the bundle
object itself and, if running with a security manager, the appropriate
OSGi AdminPermission(s).

If we adopt the "Access to class files and resources" requirement which I
suggested in the "exporting things" thread then an OSGi implementation
built on top of the platform module system could use that to expose the
resources of a module artifact as needed.

>>> Java EE's requirements are much simpler and are satisfied by
>>> module-granularity dependencies and (in certain cases) resource importing.
>> 
>> Aside from things like deployment descriptors, which are interpreted by
>> Java EE implementations, does the EE specification require EE "modules"
>> to export resources to other modules?
> 
> The Java EE 7 platform specification does seem to do so, with language 
> such as "Components in the web container may have access to the 
> following classes and resources. [...] The classes and resources 
> accessible to any other web modules included in the
> same ear file, as described above."
> 
> The spec also seems to use the term "contents" interchangeably with 
> "classes and resources".

The Java EE specification, just like OSGi, is explicitly built on the JAR
file format and derivations thereof, so I think the same solution applies
for Java EE: An EE implementation can access a module's artifact and
expose the artifact's resources as needed.

- Mark