Proposal: #NonHierarchicalLayers (+ #LayerPrimitives)

[Thomas Watson]

> From: mark.reinhold at oracle.com
> > The addPackage method would be beneficial for other reasons when 
adhering 
> > an existing module system into a JPMS Layer.
> > 
> >  - addPackage would allow existing module systems to avoid aggressive 
> > discovery of all private packages for its own modules when building a 
JPMS 
> > ModuleDescriptor.  For the existing module systems I know about the 
class 
> > loader is used extensively as the module primitive.  Requirements wire 
up 
> > the class loaders for proper class loader delegation and typically the 

> > APIs are declared capabilities to indicate what the modular class 
loaders 
> > can load and expose to another module class loader.  But nowhere was 
it 
> > required to aggressively discover all private packages and resources 
up 
> > front.  Having an addPackage would allow such module systems to grow 
the 
> > list of private packages lazily as it is defining classes in the 
packages 
> > that are private to the module.
> 
> Understood, but for context: I previously suggested that you could use
> the `Private-Package` manifest header added by bndtools, when present,
> in order to avoid scanning a bundle for private packages [1].  In reply
> you wrote that you planned to use that approach for bundles built by
> bndtools, and that "even for the bundles where we have to scan for all
> packages the framework can easily cache the private-package information
> so that upon restart the scan is not needed again" [2].  So, I see how 
an
> `addPackage` method could help OSGi interoperation, but it doesn't 
appear
> to be a hard requirement.
> 
> (As a side note, scanning a JAR file to determine its private packages
>  is pretty fast these days, even without caching.  That's what we do for
>  automatic modules in the JPMS implementation, and so far it hasn't been
>  a significant performance problem.)

The Private-Package cannot be relied on from BND.  A very large number of 
existing bundles are not built with BND.  The option to have 
Private-Package header in BND can be turned off.  In many cases the 
exhaustive scan of the jars will be needed.  As a data point, to scan my 
current Eclipse installation (~360 bundles at well over 310 MB of jars) it 
is taking over 1000 ms to do the full scan (on a pretty new MacBook Pro).

Some optimization may speed this up.  In OSGi the artifact the bundle is 
stored in on disk is abstracted away.  There is no independent way to get 
hold of the jar file directly.  I am using OSGi APIs to do the scanning of 
the bundle wiring.  It is possible this is not as speedy as it can be, but 
the cost will always be measurable and it will linear O(n) with the number 
of bundles installed.  Without caching the results this also pays an 
upfront cost of having to open each and every jar file.  In Equinox we 
delay the opening of the jars until a class or resource is actually going 
to be loaded from the bundle.  This upfront scanning forces us to open all 
installed bundle jars every startup even though no classes or resources 
may be loaded from the jar.

The caching comes with its own price in code complication and performance 
cost to load.  Doing this lazily would greatly simplify this and should 
result in better performance for really large installations.  We have 
common cases with 1000s of bundles installed.


> 
> > - In my prototype I also have a boot strapping issue that addPackage 
would 
> > be helpful for.  In my prototype I have a launcher that assembles a 
module 
> > Layer which then holds the OSGi framework implementation.  In this 
case 
> > the launcher is using Java 9 Layer API and then is loading up a 
standard 
> > OSGi framework implementation using only OSGi standard API.  The 
launcher 
> > then assigns the framework implementation a ModuleDescriptor to 
represent 
> > the OSGi framework.  In OSGi the framework is represented by an OSGi 
> > bundle called the "system.bundle".  So in this case the 
ModuleDescriptor 
> > also has a name "system.bundle".  And the ModuleDescriptor has a well 
> > known list of OSGi APIs it exports.  But the launcher does not know 
the 
> > private implementation packages of the framework implementation.  In 
my 
> > prototype I just hack in the list of private package for the equinox 
> > framework, but this is far from ideal.
> 
> Aren't the solutions for the previous problem also applicable here?  If
> such a launcher must be able to launch an arbitrary OSGi framework
> implementation then can't it use the `Private-Package` manifest header,
> when available, or else scan the framework's bundle for private packages
> and cache that result for later use?

Perhaps, but there is still a complication with system bundle fragments. I 
mentioned this in a past thread about how important it is to support 
dynamic fragment attachment.  In general I don't think there is a large 
number of OSGi users depending on dynamic fragment attachment to normal 
host bundles.  But system bundle fragments are a different story.  These 
are dynamically attached to the framework implementation class loader. 
These require special support by the launcher in order to provide the 
framework the ability to add content to its own class loader.

In fact, this is how my JPMS POC is implemented, in its own system bundle 
framework fragment.  To make this work the list of equinox framework 
packages and the list of packages for framework fragment must be specified 
for the framework module descriptor by the launcher.  For my POC I just 
hardcoded these packages directly into the launcher.  In the general case 
the launcher will not know what framework fragments will be installed 
during the lifetime of the framework instance.  The addPackage and 
addExports method would allow such a launcher to handle when new packages 
are added from a framework fragment.

> 
> A higher-level question is, how important is it for an OSGi-on-JPMS
> launcher to be completely independent of the OSGi framework that it's
> launching?  If every framework provides its own such launcher then, in
> each one, a little bit of build logic could be used to embed a list of
> the system bundle's private packages in the launcher itself, so that
> there's no need to duplicate that list manually.

There are framework independent launchers that only use OSGi APIs (BND 
tools has one such example).  It would be beneficial if these launchers 
could do the right thing when running on JPMS.  The addPackage and 
addExportToAll method would make this possible.

> 
> > I did say that I could live without supporting dynamic attachment of 
> > fragments which provided additional packages.  But I do not want that 
> > compromise from me to justify not adding the method addPackage to the 
> > controller.  I think it would be beneficial to existing module systems 

> > that need to live within a proper JPMS Layer. 
> 
> Understood.
> 
> > It would help to understand what complications addPackage bring to 
JPMS 
> > above the other dynamic add methods already on the Module API and the 
> > others being proposed for the Controller API here.
> 
> My concern is less about how these additional methods would complicate
> JPMS -- although they would -- than about the constraints they would
> place on the future evolution of JPMS, the rest of the platform, and its
> implementations.
> 
> The `addReads` and `addOpens` methods that we've already added to the
> `Layer.Controller` class require essentially the same support from the
> run-time library and the JVM as the corresponding methods in the 
`Module`
> class.  Those methods in the `Module` class are essential to the design,
> so adding them to the `Controller` class presented little risk.
> 
> The same argument can be made for the qualified `addExports` method and
> the `addUses` method, though whether adding just these two methods to 
the
> `Controller` API would benefit anyone is unclear.
> 
> The main problem with the remainder of the proposed methods is that they
> vastly increase the space of situations in which the run-time system 
must
> be prepared to update the definitions of modules.  That can, in turn,
> limit both the space of potential implementations and the bounds of
> practical performance.

I obviously don't know the internal details of the java implementation, I 
can understand your potential concerns.  I do wonder if the performance 
improvements you speak of can still be done in the vast majority of cases 
where the controller is not exposed at all.  The controller is only made 
available when using one of the static defineModule methods on the Layer 
API.  The static defineModule methods are useful for cases where an 
existing module system is trying to adhere their modules into JPMS and the 
additional dynamics are needed by the controller.  But for the typical 
case where it is JPMS only there is no need for the controller at all, so 
you should be able to count on things remaining static.

I say this knowing it does not simplify the spec or RI for the controller 
case, but it should allow for you to optimize the common case more at some 
point.

> 
> The current, internal versions of these methods are used only in limited
> circumstances.  Some are invoked only in the process of instantiating a
> module, and never thereafter.  Others exist only to support white-box
> testing, and are never invoked during normal run-time operations.  These
> limitations on use simplify both the specification and the 
implementation
> of the rest of the system, which can be written under the assumption 
that
> module definitions change in these ways only rarely, if ever.
> 
> To generalize these methods and expose them for use at arbitrary times
> would add complexity to both the specification and the RI, and would 
very
> likely degrade performance.  To make these methods part of the Java SE
> API would -- more importantly -- mandate that all future specifications
> and implementations support such updates to module definitions at
> arbitrary times.
> 
> It is not necessary to expose these methods in order to achieve the goal
> of this JSR.  Designing such low-level primitives via which different
> module systems can interoperate on an intimate basis was never a goal
> of this JSR.  To avoid adding complexity to the specification and its
> implementations in the short term, and to impose as few constraints as
> possible on the evolution of the specification and its implementations
> in the long term, these methods should remain internal.
> 
> - Mark
> 
> 
> [1] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-
> October/000409.html
> [2] http://mail.openjdk.java.net/pipermail/jpms-spec-comments/2016-
> October/000070.html
>