Interoperation (Was: Re: Module-system requirements)

[mark.reinhold at oracle.com]

2015/2/16 5:57 -0800, david.lloyd at redhat.com:
> On 02/15/2015 07:24 PM, mark.reinhold at oracle.com wrote:
>> 2015/2/11 11:56 -0800, david.lloyd at redhat.com:
>>> I'm concerned that the interoperation point doesn't quite go far enough.
>>> It says "It must be possible for another module system, such as OSGi,
>>> to locate Java modules and resolve them using its own resolver, except
>>> possibly for core system modules."
>>> 
>>> I agree with this requirement, such as it is, but I would also assert
>>> that it should be possible to retrofit not only OSGi but Java EE, Maven,
>>> and even legacy "modules" like extensions and so on as modules.  The
>>> reason I make this assertion is that we have done all of these things
>>> with success and (I dare say) relative ease, even to the extent of
>>> achieving both EE and OSGi certification with such a retrofit.
>> 
>> Can you explain what you mean, exactly, by "retrofit"?
> 
> We have implemented both a certified OSGi implementation as well as a 
> certified Java EE implementation which maps these specifications' module 
> concepts onto our own concept of a module.  ...
> 
>> Are you saying that the JBoss module system can consume Java EE WAR
>> files, OSGi bundles, and Maven projects, as they stand and without
>> change?
> 
> Yes*.  This is because of implementation choices we've made - each 
> module is a class loader, and we can carefully control which classes 
> *and* resources are exported from one to another.  That said, I do think 
> it's still worth exploring a different approach than ours regarding 
> conceptually combining resource directories and packages, which is what 
> we do today and is not without disadvantage.
> 
> ...
> 
>>> ...
>>> 
>>> To achieve this additional requirement does require a few very specific
>>> implementation considerations but I do not believe that any of these
>>> considerations contraindicate any of the other requirements stated here.
>> 
>> Can you be more specific as to what those considerations are?  Can they
>> be expressed as module-system requirements?
> 
> In the case of both Java EE and OSGi, I think it's safe to say that you 
> need to apply a strict 1:1 mapping between modules and class loaders for 
> this to work.

I don't think we can mandate this.  To divide the SE platform into
modules in a compatible way requires the ability to load multiple modules
via the same class loader, so as to respect long-standing assumptions
about which platform classes are loaded by the built-in class loaders
(i.e., the bootstrap and extension loaders).  In an early Jigsaw
prototype we tried the 1:1 module/loader mapping for platform modules
and found it unworkable from a compatibility perspective.

Aside from the SE platform itself, I don't think we can force all
applications to adopt the unique-class-loader-per-module approach.
Oracle's WebLogic application server, e.g., doesn't do that today and
it'd be difficult, if not impossible, to modify it to do so without
breaking existing applications.

I suggest the module system be as neutral as possible with respect to how
modules and class loaders are related.  A complex application should be
able to spin up a class loader for every module it loads, if needed
(excepting non-upgradeable platform classes), but not all applications
should be forced to do so.

New requirement, for the Dynamic Configuration section:

  - _Control of class loaders_ --- An application that loads a module
    into a dynamic configuration must be able to specify the class loader
    to be used to load that module.  That loader need not be the same as
    the loader used to load any other module in the configuration.

>                Also you need an API to allow run-time systems to create 
> their own configurations (we call them "module loaders", akin in purpose 
> and function to class loaders) and assemble modules from resources 
> (JARs) and dependency specifications, and such APIs have to be able to 
> relink modules or deregister modules to allow them to be GC'd (though 
> the boot module loader in our case does not expose relink or unload 
> functionality for obvious reasons).

These needs are already captured, I think, by the existing requirements
related to dynamic configurations.

> OSGi specifically requires that bundle class loaders implement an OSGi 
> spec interface, implying that the module class loader class must be 
> extensible (as I said a couple of years back though, we make most of our 
> methods final on the module class loader to disallow any really bizarre 
> behavior, and this works well in practice).

Which OSGi interface are you thinking of here?

>                                              We also had to implement a 
> pre- and/or post-class-define hook per module to allow some OSGi magic 
> to happen.

These are non-final protected methods in your module class loader?

Could you forward a description of these methods to the list?

>             Finally there is a per-module fallback resource loader that 
> our OSGi implementation used for one particular part of the spec.

Which part, in particular?

To what extent does OSGi assume that bundles "export" their resources?
If we decide that a module's resources are not available outside the
module itself then will the approach of extending a module class loader
to be an OSGi bundle loader even work?

> Java EE's requirements are much simpler and are satisfied by 
> module-granularity dependencies and (in certain cases) resource importing.

Aside from things like deployment descriptors, which are interpreted by
Java EE implementations, does the EE specification require EE "modules"
to export resources to other modules?

> The APIs to do these things are all pretty simple though.
> 
>>> Thus I feel this requirement should be amended to something along the
>>> lines of:
>>> 
>>> Interoperation - It must be possible for another module system, such as
>>> OSGi, to locate Java modules and resolve them using its own resolver,
>>> except possibly for core system modules.  It must be possible for other
>>> module systems to reuse the framework provided by this specification in
>>> order to facilitate a standard, well-defined mechanism by which
>>> different present and future module systems may interoperate.
>> 
>> Unless I've misunderstood something, this sounds more like a research
>> project -- a "meta" module system, in fact -- which is well outside the
>> scope of this JSR.
> 
> Maybe, but I contend that a lot (enough, insofar as any amount is ever 
> enough) of research has been done, and that in any event there will have 
> to be an API which implements the functionality of this JSR anyway, and 
> it will be used as such no matter our intent so IMO it's best to use 
> what has been learned to solve the problem in the best manner possible.

I agree, but I think your proposed amendment is problematic -- it's too
difficult to test, since it's worded in terms of unspecified "present and
future module systems".  Are there other, more-specific requirements we
can identify which would enable this stronger kind of interoperation?

- Mark