Updates on Interoperability

[Bryan Atsatt]

Richard S. Hall wrote:
> Bryan Atsatt wrote:
>> Richard S. Hall wrote:
>>> Sam Pullara wrote:
>>>> Did we decide which way it will be compatible?  I'm a little 
>>>> concerned that implementing something that can use OSGi modules is 
>>>> a far bigger task that implementing something that OSGi can use.  
>>>> If we are going for total compatibility it seems like we should 
>>>> just use OSGi.  Personally I wanted something in Java that was 
>>>> simpler and cleaner that OSGi-users could leverage if they wanted.  
>>>> But I'm coming at this from the 
>>>> we-need-something-like-maven/gem/ivy camp.
>>>>
>>>> The vast majority of people that are going to use this system will 
>>>> have never heard of OSGI nor care about any of their more subtle 
>>>> features and just want an easy way to leverage the vast amount of 
>>>> libraries available (virtually all of them are not OSGi modules 
>>>> today but are jar files in maven repositories).  We shouldn't be 
>>>> specing out some sort of uber container but rather a simple way to 
>>>> tie code to its dependencies.
>>>
>>> If this is all that people wanted, then we could just define 
>>> repositories and associated metadata and forget about all runtime 
>>> modularity support, but I strongly disagree that this is all people 
>>> want or else there wouldn't be such an upsurge in OSGi interest and 
>>> adoption.
>>>
>>>> I strongly suggest KISS applies to our effort.  I write this 
>>>> knowing that the committee seems to be heavily in favor of this 
>>>> alternate view that we implement the kitchen sink.
>>>
>>> I think our interest is similar, but perhaps our conclusions are 
>>> different. Initially, I think my comments were construed as an 
>>> argument against split packages in general, but I am really arguing 
>>> the same as you (I think) that we don't really need split packages 
>>> to be a concept in our "interoperability API", if that is how we are 
>>> to view the 277 API. Of course, I am also willing to have the former 
>>> argument too, but I will leave that for another time. :-)
>>>
>>> From my point of view, I do not think it is wholly reasonable to 
>>> assume that we want to try to accomplish interoperability across 
>>> module systems, where that includes arbitrarily sharing split 
>>> packages across module systems.
>> Nor do I, but I'm trying to understand the implications if we assume 
>> that OSGi will continue to split packages.
>>>
>>> Let me try to accurately conjure the peculiarities of split packages 
>>> in the OSGi module layer...
>>>
>>> We support split packages in two ways, via required bundles (i.e., 
>>> module dependencies) and via bundle fragments. In general, we 
>>> recommend that if you split your packages that you attach mandatory 
>>> attributes on them so that importers do not get them by mistake. By 
>>> requiring a bundle, you can ignore mandatory attributes and get 
>>> everything a bundle has to offer and combine it with other required 
>>> bundles that might be exporting other parts of the split packages. 
>>> Given this, there is a semi-reasonable chance* that we can tell you 
>>> which bundles are contributing to the "complete" packages, but we 
>>> have no way of knowing if/when combined split packages are complete. 
>>> Further, we have to assume that split package parts offered by 
>>> different bundles do not overlap, because if we allowed them to 
>>> overlap then we would have shadowing and ordering issues. Again we 
>>> have no easy way to verify that they do not overlap, so we just 
>>> assume they don't.
>>>
>>> Fragments on the other hand are a little trickier, since split 
>>> packages from them are loaded from the same class loader to provide 
>>> package private access (which is not afforded for split packages in 
>>> the require bundle approach). The "host" bundle doesn't really know 
>>> about fragments and it is possible for fragments to shadow or extend 
>>> packages in the "host" bundle. Since fragments provide their split 
>>> packages in an implicit way (i.e., they largely just become part of 
>>> the host bundle's class path), there is no easy way to determine if 
>>> a fragment is contributing to a given package or not. (* This 
>>> feature of fragments also makes it so it is not really possible to 
>>> know who is contributing classes to a split package in the require 
>>> bundle case.)
>> Ok, so first I assume it would make sense for an OSGi Repository to 
>> simply never return a ModuleDefinition for a fragment.
>
> I am not sure I understand your point. Host bundles would be returned 
> and during the resolve process, fragments might be attached to that 
> host, thus modifying its content in ways that we cannot predict.
I simply meant that a fragment would never be returned by itself, as a 
separate ModuleDefinition.
>
>> And the "you don't know what you've got till it's resolved" problem 
>> is going to be present in any module system for which package name is 
>> not a required component of a dependency expression. But as long as 
>> re-export is not the default behavior (it isn't), doesn't this really 
>> just boil down to the leakage problem? And this is addressed in 277 
>> the same as it is in OSGi: you must declare such "leaks" as 
>> re-exports ("uses"). Not perfect, but probably good enough.
>>
>> So I think we're left with the issue of a single search visiting 
>> multiple loaders that can return the same package. This won't happen 
>> in the fragment case, since the "split" gets mended at runtime, but 
>> it will in the require-bundle case. And this will manifest as 
>> multiple entries in the import list which export the same package.
>>
>> But even this isn't an issue, as long as dependency resolution relies 
>> solely on declared exports and re-exports of a module, and not on 
>> what is available by browsing the imports of that module.
>>
>> So I'm back to my original statement that we just need to acknowledge 
>> that package duplication may exist in an import list.
>>
>> Or am I missing something here?
>
> I am just not sure why we want to make the fact that some module 
> systems support split packages a visible concept in our 
> "interoperability API", when it would be difficult to get 
> interoperability around split packages across module systems when we 
> already have issues with split packages within a single module system. 
> Especially, given my description above, since we would not easily know 
> which modules were contributing to a split package in OSGi.
I was just exploring whether or not we need to make it a visible concept 
in order for cross module system resolution to work correctly. You may 
note in Stanley's doc that he simply took the position that a JAM 
definition would fail to resolve against an OSGi definition that split 
packages. This may be an acceptable solution, but it still requires a 
means to *detect* that a split has occurred.
>
> -> richard
>
>>>
>>> I think that is fairly complete description of the situation, sorry 
>>> if it is somewhat terse. In summary, I wouldn't assume that we can 
>>> get reasonable interoperability at this level. If you need these 
>>> types of features, then you should stick to a single modularity 
>>> framework...preferably OSGi. ;-)
>>>
>>> But my overall point is, I believe that if we can support 
>>> interoperability at module- and package-level dependencies across 
>>> module systems, then I think we would hit most use cases.
>>>
>>> -> richard
>>>
>>>>
>>>> Sam
>>>>
>>>> On Apr 25, 2008, at 1:57 PM, Richard S. Hall wrote:
>>>>
>>>>> Bryan Atsatt wrote:
>>>>>> Richard S. Hall wrote:
>>>>>>> Gordon Hirsch wrote:
>>>>>>>>> 1. Map its dependency declarations into a standard runtime 
>>>>>>>>> representation.
>>>>>>>>> 2. Support a standard search model over its stored modules, 
>>>>>>>>> using the runtime dependency expressions.
>>>>>>>>> 3. Map its stored module data into a standard runtime 
>>>>>>>>> representation that can be returned by the search.
>>>>>>>>
>>>>>>>> One challenge lies in defining the "standard runtime 
>>>>>>>> representations" and "standard search model" in a universal 
>>>>>>>> enough way to encompass OSGi and other module systems. This 
>>>>>>>> implies embracing concepts (in these standard representations 
>>>>>>>> and search model) that were not universally liked by the EG 
>>>>>>>> early on. (Split packages and package-level import/export come 
>>>>>>>> to mind.)
>>>>>>>
>>>>>>> Package-level import/export seem like a must, but I still don't 
>>>>>>> see split packages as a necessity.
>>>>>> Strongly agreed on import-by-package, and that ModuleDefinition 
>>>>>> requires a method to enumerate exported packages (and probably an 
>>>>>> exportsPackage(String packageName) method to enable an efficient 
>>>>>> Query).
>>>>>> On the split package front, however, it seems a little fuzzy to 
>>>>>> me. If an OSGi implementation of 277 is also going to remain OSGi 
>>>>>> Rx compliant, it will still need to support split packages, 
>>>>>> right? If so, don't we need to surface this fact at the 277 
>>>>>> level? Perhaps that is as simple as acknowledging that 
>>>>>> Module.getImportedModules() may return more than one instance 
>>>>>> that exports a given package.
>>>>>
>>>>> That all depends on if the 277 API is a subset of OSGi 
>>>>> functionality or equal to it, I suppose.
>>>>>
>>>>> -> richard
>>>>>>
>>>>>> // Bryan
>>>
>