Relationship to JSR 291 [was: Re: Bryan's comments]

[Bryan Atsatt]

Richard S. Hall wrote:
> Bryan Atsatt wrote:
>> Stanley M. Ho wrote:
>>> Glyn Normington wrote:
>>>>
>>>> *Bryan Atsatt <bryan.atsatt at ORACLE.COM>* wrote on 30/05/2007 07:57:59:
>>>> ...
>>>>  > So the open issue is the richness of the import "language": must we
>>>>  > support only lowest-common-denominator, or can we do better without
>>>>  > over-complicating the design?
>>>>  >
>>>>  > I for one would like to be able to have a single module express
>>>>  > dependencies on modules from both the same and different
>>>> ModuleSystems,
>>>>  > *using the standard semantics of each*. This may be reaching too
>>>> far,
>>>>  > but we should at least explore it seriously while we figure out what
>>>>  > interop means here...
>>>>
>>>> At this point, I feel that is likely to be reaching too far, but I'm
>>>> happy to play along and see what we can learn along the way.
>>>
>>> I agreed with Glyn that this might be reaching too far. Before we dive
>>> too much into how to implement interoperability, I think one of the
>>> outstanding questions we should answer first is what degree of import
>>> interoperability we want to offer. There are four possibilities:
>>>
>>> 1. JSR 277 module imports OSGi module by module name
>>> 2. OSGi module imports JSR 277 module by module name
>>> 3. OSGi module imports JSR 277 module by package name
>>> 4. JSR 277 module imports OSGi module by package name
>>>
>>> Let's ignore the 294 issues and the module initialization issues for now
>>> to simplify this discussion.
>>>
>>> I think we all agreed #1 and #2 are important to support and the
>>> reflective APIs already enable these (of course, there are minor issues
>>> we still have to address as we evolve the APIs.)
>>>
>>> That said, it is unclear to me how important it is to support #3 and #4,
>>> so I think the first question for this EG is whether we want to support
>>> #3 and #4 at all. However, let's pretend #3 and #4 are important for
>>> this specific discussion.
>>>
>>> I think #3 is already possible since the OSGi framework could look up
>>> the appropriate module which exported the package in the repository,
>>> using the reflective APIs. Afterwards, the OSGi framework can then do
>>> the necessary wiring. Yes, the reflective APIs can be further refined to
>>> make it easier for the OSGi framework to query modules by exported
>>> package name from the repository, but this is a minor issue that we can
>>> address easily.
>>>
>>> #4 is a bit complicated, because there is no import-by-package semantic
>>> in the current module layer's APIs and I think we all agreed that we
>>> don't want to support this semantic directly in the module system
>>> defined by JSR 277.
>>
>> Not me.
>>
>> I do understand your point that this is more difficult, but I don't
>> really care--it is our job to do the Right Thing here. Sure, if it is
>> not feasible in the time frame we have, then so be it. But we haven't
>> even tried!
>>
>> If we're forced to make a choice between the two models, I'd pick
>> import-by-package with zero hesitation.
>>
>> OSGi has a lot of experience here, and we shouldn't ignore it.
>> Import-by-name was added late in the game, and is now seriously
>> downplayed, for good reason.
>
> Just for information, import-by-name (or module dependencies) was not
> added to OSGi R4 because they were thought to be widely needed or
> useful, adding them was always somewhat contentious which is why the R4
> spec contained a section saying why you shouldn't use them (this section
> has been further expanded in R4.1). The main motivation for adding them
> was certainly legacy situation and for tightly coupled sub-systems.
>
> All in all, I agree with everything that Bryan is saying and I argued
> these same points early on; however, I was under the impression that we
> had long since lost that battle.
>
> Bryan, it is unclear to me whether your view is that we need to make
> sure that 277 supports other modules systems the use import-by-package
> or if you think that 277 should somehow directly support
> import-by-package (i.e., explicitly expose such concepts).

I want 277 to explicitly expose the concept of import-by-package through
appropriate APIs.

It would be very nice if the 277ModuleSystem implementation itself
directly supports import-by-package (case #3). But this is less critical
to me than case #4...

I want a 277ModuleDefinition to be able to import-by-package an
OSGiModuleDefinition (i.e. a bundle exposed via the 277 APIs). I believe
this case will be a particularly important one. By the time SE 7 ships,
there will be a very large number of OSGi bundles in the world, and near
zero 277 modules.

If 277 modules want to have a chance to survive in that world, they will
*need* to rely on OSGi bundles. Could they rely only on import-by-name?
Sure. But I believe natural selection would eventually result in the
extinction of 277 modules, in favor of systems that are less brittle.

I actually like the development/deployment/API model in 277, and want to
use it. At the same time, I am acutely aware that OSGi is being widely
adopted, at an accelerating rate.

So I believe that it is incumbent on this JSR to support OSGi in a
manner that is natural to it.

// Bryan

>
> -> richard
>
>>
>> And Oracle has a lot of relevant experience as well, since the shared
>> loader mechanism I created for our AS stack ("shared libraries")
>> supports only the import-by-name model. We have *lots* of
>> shared-libraries (I've seen 100+ running simultaneously), used by many
>> different components in the stack, in addition to customer applications.
>> It is a tremendous pain when refactoring or even simple renaming has to
>> be done (I hope to shift this implementation onto 277 to solve this
>> problem, rather than inventing an interim approach). And this kind of
>> change happens far more frequently than you probably expect.
>>
>> I don't want to repeat the same mistake here. Nor do I think that
>> leaving a long trail of extraneous "view" modules behind for
>> compatibility is a good solution.
>>
>> Worse still, if the 277 APIs don't support import-by-package, then the
>> compiler won't support it either. If the compiler doesn't support it,
>> then we've lost a golden opportunity to move to a better world, one in
>> which packaging is irrelevant.
>>
>> We're supposed to be eliminating "jar-hell" here. Let's not simply
>> replace it with "module-hell".
>>
>>> In this context, the question is really about what
>>> it would take to make #4 possible if we want to support it, and so far
>>> there are two different approaches we have discussed:
>>>
>>> a. Treat import-by-package separately from import-by-module in the
>>> reflective APIs and make the module layer fully aware of the
>>> import-by-package concept.
>>>
>>> b. Expose exported-package using the ModuleDefinition abstraction, and
>>> provide necessary hooks (e.g. consistency checking) for the module
>>> system to resolve this kind of dependency appropriately. The module
>>> layer is not aware of the import-by-package concept at all. New import
>>> dependency granularity could also be introduced in a similar manner in
>>> the future without requiring significant changes in the module layer.
>>>
>>>
>>> As Richard and I discussed in previous emails, it seems to make the most
>>> sense for JSR 277 to define the minimal set of features possible in the
>>> module layer to accomplish what needs to be accomplished to provide core
>>> modularity support in the Java platform. I think (b) follows this
>>> principle nicely while (a) does not. If we decide to support #3 and #4,
>>> then how we want to address #4 would impact our overall design
>>> significantly.
>>>
>>> Is there anyone in the EG provide good reasons why #3 and #4 are
>>> important to be supported? If so, is there anyone in the EG think (b) is
>>> not feasible at all and we should do (a) instead to support #4? Can you
>>> explain your rationale?
>>
>> Yes, and yes. See above for rationale (ease-of-refactoring and compiler
>> support).
>>
>>>
>>>
>>>>  > RESOLUTION MODELS
>>>>  >
>>>>  > The current design requires that each ModuleSystem provide its own
>>>>  > resolution logic, and that each definition will be resolved by its
>>>>  > owning ModuleSystem. This model appears to provide flexibility for
>>>>  > significant differences in implementation, but we really don't know
>>>>  > enough at this point. Perhaps only an actual second implementation
>>>> will
>>>>  > tell us if this provides useful flexibility.
>>>
>>> In the existing module systems we are aware, e.g. OSGi and NetBeans, we
>>> already know their resolution logics are different from each other. Even
>>> if they are migrated/re-based to JSR 277 in the future, I think we
>>> should expect their resolution logic would remain the same to maintain
>>> backward compatibility with their existing modules/bundles. In other
>>> words, their resolution logic would still be different from each other.
>>
>> And this is very likely to be the source of subtle but very painful
>> interop issues. Which is why I also suggested that we should consider
>> moving to a canonical, though evolvable, resolution mechanism.
>>
>>>
>>>> A phased approach would be particularly beneficial if the initial phase
>>>> could be delivered as part of Java 7 and subsequent phases implemented
>>>> strictly on top of Java 7. But getting the API right up front might be
>>>> tricky unless we can spot some really good abstractions or prototype
>>>> the
>>>> later phases sufficiently well. Is that the kind of phasing you had in
>>>> mind?
>>>
>>> Once an API is put into the SE platform, it will stick around almost
>>> forever. I think if we go with the phased approach on interoperability
>>> support (I believe this is what Bryan suggested), it would be much
>>> better to simply left out those features/APIs which do not have
>>> implementations to back them up from JDK 7, rather than putting the
>>> half-baked features/APIs into the release and regret later. ;) We can
>>> always add new APIs in subsequence releases if we decide to offer better
>>> degree of interoperability in the future.
>>
>> I couldn't agree more that we don't want to put half-baked APIs into SE!
>> If we cannot expect significant validation of our interoperation model
>> (whatever we settle on) from NetBeans and OSGi, then we shouldn't expose
>> these APIs at all. Period.
>>
>> But this doesn't just apply to import-by-package. We haven't even
>> defined how class and resource sharing will occur. I won't be a bit
>> surprised if we want one or more new types here.
>>
>>
>> And, last thoughts for the day...
>>
>> Even if you are adamantly opposed to import-by-package, please take a
>> good look at the other ideas buried in my message from yesterday,
>> particularly the ModuleContext proposal.
>>
>> It is completely independent of import-by-package, and seems to me at
>> least to be a concise, flexible model for caching/isolating/finding
>> Module instances. (The idea of exposing a resolve() method on it also
>> makes a clean API, even if it delegates to ModuleSystem instead of a new
>> resolver type.)
>>
>> There are other potentially interesting tidbits there as well
>> (PlatformBinding exposure on definition, Query support for current
>> platform, ImportDependency *as* Query).
>>
>> I'd hate for the baby to be thrown out with the bath water :^)
>>
>> // Bryan
>>
>>>
>>> - Stanley
>>>
>