Interoperation (Was: Re: Module-system requirements)

[David M. Lloyd]

On 03/09/2015 03:54 PM, mark.reinhold at oracle.com wrote:
> 2015/2/16 5:57 -0800, david.lloyd at redhat.com:
>> On 02/15/2015 07:24 PM, mark.reinhold at oracle.com wrote:
>>> 2015/2/11 11:56 -0800, david.lloyd at redhat.com:
>>>> I'm concerned that the interoperation point doesn't quite go far enough.
>>>> It says "It must be possible for another module system, such as OSGi,
>>>> to locate Java modules and resolve them using its own resolver, except
>>>> possibly for core system modules."
>>>>
>>>> I agree with this requirement, such as it is, but I would also assert
>>>> that it should be possible to retrofit not only OSGi but Java EE, Maven,
>>>> and even legacy "modules" like extensions and so on as modules.  The
>>>> reason I make this assertion is that we have done all of these things
>>>> with success and (I dare say) relative ease, even to the extent of
>>>> achieving both EE and OSGi certification with such a retrofit.
>>>
>>> Can you explain what you mean, exactly, by "retrofit"?
>>
>> We have implemented both a certified OSGi implementation as well as a
>> certified Java EE implementation which maps these specifications' module
>> concepts onto our own concept of a module.  ...
>>
>>> Are you saying that the JBoss module system can consume Java EE WAR
>>> files, OSGi bundles, and Maven projects, as they stand and without
>>> change?
>>
>> Yes*.  This is because of implementation choices we've made - each
>> module is a class loader, and we can carefully control which classes
>> *and* resources are exported from one to another.  That said, I do think
>> it's still worth exploring a different approach than ours regarding
>> conceptually combining resource directories and packages, which is what
>> we do today and is not without disadvantage.
>>
>> ...
>>
>>>> ...
>>>>
>>>> To achieve this additional requirement does require a few very specific
>>>> implementation considerations but I do not believe that any of these
>>>> considerations contraindicate any of the other requirements stated here.
>>>
>>> Can you be more specific as to what those considerations are?  Can they
>>> be expressed as module-system requirements?
>>
>> In the case of both Java EE and OSGi, I think it's safe to say that you
>> need to apply a strict 1:1 mapping between modules and class loaders for
>> this to work.
>
> I don't think we can mandate this.  To divide the SE platform into
> modules in a compatible way requires the ability to load multiple modules
> via the same class loader, so as to respect long-standing assumptions
> about which platform classes are loaded by the built-in class loaders
> (i.e., the bootstrap and extension loaders).  In an early Jigsaw
> prototype we tried the 1:1 module/loader mapping for platform modules
> and found it unworkable from a compatibility perspective.

I went through the jigsaw list archives looking for this discussion and 
didn't turn up anything; was this discussed on-list and I am just 
failing to find it?

> Aside from the SE platform itself, I don't think we can force all
> applications to adopt the unique-class-loader-per-module approach.
> Oracle's WebLogic application server, e.g., doesn't do that today and
> it'd be difficult, if not impossible, to modify it to do so without
> breaking existing applications.

I guess I can't really comment on what WebLogic does or does not do.  I 
can say however that we were able to move from a flat classpath 
architecture to a modular architecture like this without breaking things 
due to the modularization.  The Java EE spec is pretty clear about what 
needs to be visible to what, and by translating this information into 
module dependency information, we were able to make a smooth transition. 
  I can't really imagine an application that would be broken unless they 
were relying on some specific non-spec behavior as the EE spec says:

"[...] A Java EE application should not assume that all components in 
the application will be available on the class path of the application 
at run time. Each component might be loaded into a separate class loader 
with a separate namespace. [...]"

Note that in the EE spec, the term "component" has a finer granularity 
than "module", i.e. there are 1..N components per module, whereas an 
"application" can contain 1 or more modules.

At the same time though, most of the language around class loaders uses 
"may" so it's all pretty flexible in terms of implementation requirements.

> I suggest the module system be as neutral as possible with respect to how
> modules and class loaders are related.  A complex application should be
> able to spin up a class loader for every module it loads, if needed
> (excepting non-upgradeable platform classes), but not all applications
> should be forced to do so.
>
> New requirement, for the Dynamic Configuration section:
>
>    - _Control of class loaders_ --- An application that loads a module
>      into a dynamic configuration must be able to specify the class loader
>      to be used to load that module.  That loader need not be the same as
>      the loader used to load any other module in the configuration.

This goes part of the way, for sure.  I'm not 100% sold yet though. 
It's not easy to let go of something that (from our perspective) is 
tried and true.

>>                 Also you need an API to allow run-time systems to create
>> their own configurations (we call them "module loaders", akin in purpose
>> and function to class loaders) and assemble modules from resources
>> (JARs) and dependency specifications, and such APIs have to be able to
>> relink modules or deregister modules to allow them to be GC'd (though
>> the boot module loader in our case does not expose relink or unload
>> functionality for obvious reasons).
>
> These needs are already captured, I think, by the existing requirements
> related to dynamic configurations.
>
>> OSGi specifically requires that bundle class loaders implement an OSGi
>> spec interface, implying that the module class loader class must be
>> extensible (as I said a couple of years back though, we make most of our
>> methods final on the module class loader to disallow any really bizarre
>> behavior, and this works well in practice).
>
> Which OSGi interface are you thinking of here?

In OSGi Core 6.0.0 section 3.9.9:

"Class Loader - An OSGi framework must ensure that the class loader of a 
class that comes from
a bundle implements the BundleReference interface."

The BundleReference interface has a single getBundle() method on it.

>>                                               We also had to implement a
>> pre- and/or post-class-define hook per module to allow some OSGi magic
>> to happen.
>
> These are non-final protected methods in your module class loader?
>
> Could you forward a description of these methods to the list?

Yeah, they look like this:

     protected void preDefine(ClassSpec classSpec, String className) {
     }

     protected void postDefine(ClassSpec classSpec, Class<?> definedClass) {
     }

ClassSpec contains the bytes and CodeSource of the class.  The hooks 
don't have opportunity to do anything like rewrite the class bytes, but 
we also accept a java.lang.instrument.ClassFileTransformer instance in 
the module definition for that purpose (though we aren't using it at 
this point).  Our OSGi implementation only ended up using one of these 
hooks (preDefine I think).  They were added to support lazy bundle 
activation.

>>              Finally there is a per-module fallback resource loader that
>> our OSGi implementation used for one particular part of the spec.
>
> Which part, in particular?

I'll have to go back through my notes to say for sure, but I think we 
used it for a couple of purposes:

1) To support dynamic imports: https://issues.jboss.org/browse/MODULES-30
2) As a general hook to allow for OSGi-style "lazy" packages (the loader 
would examine the package of the class in question, recalculate the 
package set, and then retry the class load): 
https://issues.jboss.org/browse/MODULES-63

> To what extent does OSGi assume that bundles "export" their resources?
> If we decide that a module's resources are not available outside the
> module itself then will the approach of extending a module class loader
> to be an OSGi bundle loader even work?

This I will defer to an OSGi expert.  My understanding is that OSGi 
largely operates on a package basis only, but I don't know how resources 
within the packages fit in to this picture (if at all).

>> Java EE's requirements are much simpler and are satisfied by
>> module-granularity dependencies and (in certain cases) resource importing.
>
> Aside from things like deployment descriptors, which are interpreted by
> Java EE implementations, does the EE specification require EE "modules"
> to export resources to other modules?

The Java EE 7 platform specification does seem to do so, with language 
such as "Components in the web container may have access to the 
following classes and resources. [...] The classes and resources 
accessible to any other web modules included in the
same ear file, as described above."

The spec also seems to use the term "contents" interchangeably with 
"classes and resources".

>> The APIs to do these things are all pretty simple though.
>>
>>>> Thus I feel this requirement should be amended to something along the
>>>> lines of:
>>>>
>>>> Interoperation - It must be possible for another module system, such as
>>>> OSGi, to locate Java modules and resolve them using its own resolver,
>>>> except possibly for core system modules.  It must be possible for other
>>>> module systems to reuse the framework provided by this specification in
>>>> order to facilitate a standard, well-defined mechanism by which
>>>> different present and future module systems may interoperate.
>>>
>>> Unless I've misunderstood something, this sounds more like a research
>>> project -- a "meta" module system, in fact -- which is well outside the
>>> scope of this JSR.
>>
>> Maybe, but I contend that a lot (enough, insofar as any amount is ever
>> enough) of research has been done, and that in any event there will have
>> to be an API which implements the functionality of this JSR anyway, and
>> it will be used as such no matter our intent so IMO it's best to use
>> what has been learned to solve the problem in the best manner possible.
>
> I agree, but I think your proposed amendment is problematic -- it's too
> difficult to test, since it's worded in terms of unspecified "present and
> future module systems".  Are there other, more-specific requirements we
> can identify which would enable this stronger kind of interoperation?

That's fair, I suppose.  I think I would want to explore the 
compatibility implications of changing the module to class loader 
relationship before I come back with something stronger/more specific.

-- 
- DML