Proposal: #NonHierarchicalLayers (+ #LayerPrimitives)

[mark.reinhold at oracle.com]

2017/3/6 13:46:20 -0800, tjwatson at us.ibm.com:
> 2017/03/06 12:35:49 -0800, mark.reinhold at oracle.com:
>> 
>> ...
>> 
>> Thanks for the additional background, but this still sounds like a
>> performance problem that can be solved at a higher level rather than
>> in the JVM itself.
>> 
>> (Perhaps, in the long run, a future revision of the OSGi specification
>>  should add an API to support the caching of bundle package lists.)
> 
> I have my doubts the Private-Package header will become anything standard 
> in OSGi. Similar to how I would not like to list private packages in a 
> module-info.java file, I would not like having to specify the private 
> package list in the bundle manifest.
> 
> Of coarse that is not really what you are suggesting.

No, I think it's an idea worth considering.  As an optimization it's
not clear that it needs to be part of the OSGi standard, though maybe
that would be useful.

In JPMS we have a similar optimization based upon the `ModulePackages`
class-file attribute [1].  We don't ask people to list all packages in
`module-info.java` files, but the `jar` tool and others will compute
package lists and store them as the value of this attribute in
`module-info.class` files.  If the module system finds this attribute
then it uses its value, otherwise it scans the JAR file to determine
the full list of packages.

Isn't `Private-Package` of a similar nature?  In practice isn't it (or
can't it be) emitted by whatever tool you use to construct your manifest,
without human intervention, rather than written out by hand?  Framework
implementations would use this header when present, otherwise they'd scan
JAR files as needed (and hopefully cache the results).  I understand that
this might not be common today, but it could be enabled and encouraged
going forward.

>                                                       Instead you are 
> suggesting the OSGi wiring API could be enhanced to have something similar 
> to the ModuleDescriptor.packages() method to return a cached set of 
> package names.

That's one interpretation of my second suggestion, which could be used
alone or in combination with the `Private-Package` header.

Another form of that suggestion would be a simpler, lower-level API which
would allow a framework implementation to obtain a list of the packages
in a bundle without the framework reading the bundle's JAR file directly,
so that it could implement a package-list cache on its own.

>                 The issue is that information is not needed by the 
> framework (at least as of yet) and should not be needed by any other 
> bundle installed in the framework.  I would have a hard time proposing 
> such a method back to the specification.

Perhaps better interoperation with JPMS would be adequate justification.

>> ...
>> 
>> How critical is this general case?  How often are framework fragments
>> actually used in a way in which they cannot be identified when the
>> framework is launched?
> 
> I know this is important for Eclipse because there are many framework 
> extensions implemented by various eclipse projects as well as others. 
> ObjectTeams and aspectJ weaving projects are two that come to mind. 
> Equinox itself has something called Equinox regions which runs on any OSGi 
> framework as a framework extension.  These framework fragments almost 
> always have their own private packages and often have additional APIs that 
> are exported by the system bundle.  The way Eclipse provisions these 
> extensions depends on the ability to resolve them dynamically, without a 
> restart. 

Hmm, interesting data.

>> If I understand §3.15 of the OSGi Core R6 specification [6] correctly
>> then a framework implementation is not required to resolve framework
>> fragments (a.k.a. "extension bundles") dynamically, though it is allowed
>> to do so (see steps 2 and 3 of the lifecycle description at the bottom of
>> p. 85).  If an OSGi implementation can always restart when a framework
>> fragment is resolved or refreshed then isn't that a natural time to scan
>> the fragment's bundle for its packages, possibly caching that information
>> for later use?
> 
> Any practical framework will support attaching system.bundle fragments 
> dynamically.  I'm most familiar with Equinox and Felix, and I know both 
> these frameworks support this for system.bundle fragments.  I also know a 
> framework cannot pass the compliance tests for system.bundle fragments 
> without supporting dynamic attachment, despite the spec stating that it is 
> not required.

Sounds like there's a bug in the specification.

>> 2017/03/06 08:08:19 -0800, tjwatson at us.ibm.com:
>>> ...
>>> 
>>> There are framework independent launchers that only use OSGi APIs (BND
>>> tools has one such example).  It would be beneficial if these launchers
>>> could do the right thing when running on JPMS.  The addPackage and
>>> addExportToAll method would make this possible.
>> 
>> Yes, but could this problem be addressed in some other way?  Would modest
>> additions to the OSGi API allow a launcher to do a bit of introspection
>> on the framework being launched?
> 
> Perhaps, but for my usecases the launcher simply cannot know what will be 
> installed by a management agent later.  And to be a practical 
> implementation the framework really needs to support dynamic attachment of 
> system.bundle fragments.

So, here's an even higher-level question: Why is it necessary to make
the OSGi system bundle appear as a JPMS module in the first place?

The scheme I previously suggested [2] maps an OSGi bundle wiring to a
JPMS module and layer so that layers of JPMS modules can be resolved
against the bundle directly.  If a particular bundle will never be used
by a JPMS module, however, then there's no need to handle its wirings in
that way, since there's no reason it needs to appear to be a JPMS module.

In a mixed OSGi/JPMS system would we realistically expect a JPMS module
to require the OSGi system bundle, masquerading as a JPMS module, and use
its exported APIs?  That seems pretty unlikely.  If a component uses an
OSGi-specific API then shouldn't it just be written as an actual OSGi
bundle in the first place?

Could your proof-of-concept implementation be made to work without
putting the system bundle in a JPMS layer?  If so then it could continue
to implement dynamic framework extensions just as it does today, support
the dependence of JPMS modules upon OSGi bundles that export APIs that
aren't specific to OSGi, and yet still not require the exposure of the
low-level internal methods proposed by #LayerPrimitives.

- Mark


[1] http://cr.openjdk.java.net/~mr/jigsaw/spec/lang-vm.html#jigsaw-2.3
[2] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000410.html