Proposal: #NonHierarchicalLayers (+ #LayerPrimitives)

Mon Mar 6 02:02:44 UTC 2017

2017/2/27 17:47:25 +0000, tjwatson at us.ibm.com:
> From: mark.reinhold at oracle.com
>> From: david.lloyd at redhat.com
>>> ...
>>> 
>>> OSGi does support dynamic attachments of fragments.  The current 
>>> prototype cannot do this but on Jan. 4 Thomas expressed that being able 
>>> to add packages would enable this part of the specification. Re-linking 
>>> everything is (according to this email) an alternative that comes at the 
>>> cost of not supporting this feature.
>> 
>> True, but Thomas later recommended against complicating JPMS in order to
>> support this feature of OSGi [1].  He can live with having to re-resolve
>> bundles when fragments add API packages, which is not common and arguably
>> an anti-pattern.
> 
> The addPackage method would be beneficial for other reasons when adhering 
> an existing module system into a JPMS Layer.
> 
>  - addPackage would allow existing module systems to avoid aggressive 
> discovery of all private packages for its own modules when building a JPMS 
> ModuleDescriptor.  For the existing module systems I know about the class 
> loader is used extensively as the module primitive.  Requirements wire up 
> the class loaders for proper class loader delegation and typically the 
> APIs are declared capabilities to indicate what the modular class loaders 
> can load and expose to another module class loader.  But nowhere was it 
> required to aggressively discover all private packages and resources up 
> front.  Having an addPackage would allow such module systems to grow the 
> list of private packages lazily as it is defining classes in the packages 
> that are private to the module.

Understood, but for context: I previously suggested that you could use
the `Private-Package` manifest header added by bndtools, when present,
in order to avoid scanning a bundle for private packages [1].  In reply
you wrote that you planned to use that approach for bundles built by
bndtools, and that "even for the bundles where we have to scan for all
packages the framework can easily cache the private-package information
so that upon restart the scan is not needed again" [2].  So, I see how an
`addPackage` method could help OSGi interoperation, but it doesn't appear
to be a hard requirement.

(As a side note, scanning a JAR file to determine its private packages
 is pretty fast these days, even without caching.  That's what we do for
 automatic modules in the JPMS implementation, and so far it hasn't been
 a significant performance problem.)

> - In my prototype I also have a boot strapping issue that addPackage would 
> be helpful for.  In my prototype I have a launcher that assembles a module 
> Layer which then holds the OSGi framework implementation.  In this case 
> the launcher is using Java 9 Layer API and then is loading up a standard 
> OSGi framework implementation using only OSGi standard API.  The launcher 
> then assigns the framework implementation a ModuleDescriptor to represent 
> the OSGi framework.  In OSGi the framework is represented by an OSGi 
> bundle called the "system.bundle".  So in this case the ModuleDescriptor 
> also has a name "system.bundle".  And the ModuleDescriptor has a well 
> known list of OSGi APIs it exports.  But the launcher does not know the 
> private implementation packages of the framework implementation.  In my 
> prototype I just hack in the list of private package for the equinox 
> framework, but this is far from ideal.

Aren't the solutions for the previous problem also applicable here?  If
such a launcher must be able to launch an arbitrary OSGi framework
implementation then can't it use the `Private-Package` manifest header,
when available, or else scan the framework's bundle for private packages
and cache that result for later use?

A higher-level question is, how important is it for an OSGi-on-JPMS
launcher to be completely independent of the OSGi framework that it's
launching?  If every framework provides its own such launcher then, in
each one, a little bit of build logic could be used to embed a list of
the system bundle's private packages in the launcher itself, so that
there's no need to duplicate that list manually.

> I did say that I could live without supporting dynamic attachment of 
> fragments which provided additional packages.  But I do not want that 
> compromise from me to justify not adding the method addPackage to the 
> controller.  I think it would be beneficial to existing module systems 
> that need to live within a proper JPMS Layer. 

Understood.

> It would help to understand what complications addPackage bring to JPMS 
> above the other dynamic add methods already on the Module API and the 
> others being proposed for the Controller API here.

My concern is less about how these additional methods would complicate
JPMS -- although they would -- than about the constraints they would
place on the future evolution of JPMS, the rest of the platform, and its
implementations.

The `addReads` and `addOpens` methods that we've already added to the
`Layer.Controller` class require essentially the same support from the
run-time library and the JVM as the corresponding methods in the `Module`
class.  Those methods in the `Module` class are essential to the design,
so adding them to the `Controller` class presented little risk.

The same argument can be made for the qualified `addExports` method and
the `addUses` method, though whether adding just these two methods to the
`Controller` API would benefit anyone is unclear.

The main problem with the remainder of the proposed methods is that they
vastly increase the space of situations in which the run-time system must
be prepared to update the definitions of modules.  That can, in turn,
limit both the space of potential implementations and the bounds of
practical performance.

The current, internal versions of these methods are used only in limited
circumstances.  Some are invoked only in the process of instantiating a
module, and never thereafter.  Others exist only to support white-box
testing, and are never invoked during normal run-time operations.  These
limitations on use simplify both the specification and the implementation
of the rest of the system, which can be written under the assumption that
module definitions change in these ways only rarely, if ever.

To generalize these methods and expose them for use at arbitrary times
would add complexity to both the specification and the RI, and would very
likely degrade performance.  To make these methods part of the Java SE
API would -- more importantly -- mandate that all future specifications
and implementations support such updates to module definitions at
arbitrary times.

It is not necessary to expose these methods in order to achieve the goal
of this JSR.  Designing such low-level primitives via which different
module systems can interoperate on an intimate basis was never a goal
of this JSR.  To avoid adding complexity to the specification and its
implementations in the short term, and to impose as few constraints as
possible on the evolution of the specification and its implementations
in the long term, these methods should remain internal.

- Mark

[1] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000409.html
[2] http://mail.openjdk.java.net/pipermail/jpms-spec-comments/2016-October/000070.html