Updates on Interoperability

[Richard S. Hall]

Bryan Atsatt wrote:
> Richard S. Hall wrote:
>> Sam Pullara wrote:
>>> Did we decide which way it will be compatible?  I'm a little 
>>> concerned that implementing something that can use OSGi modules is a 
>>> far bigger task that implementing something that OSGi can use.  If 
>>> we are going for total compatibility it seems like we should just 
>>> use OSGi.  Personally I wanted something in Java that was simpler 
>>> and cleaner that OSGi-users could leverage if they wanted.  But I'm 
>>> coming at this from the we-need-something-like-maven/gem/ivy camp.
>>>
>>> The vast majority of people that are going to use this system will 
>>> have never heard of OSGI nor care about any of their more subtle 
>>> features and just want an easy way to leverage the vast amount of 
>>> libraries available (virtually all of them are not OSGi modules 
>>> today but are jar files in maven repositories).  We shouldn't be 
>>> specing out some sort of uber container but rather a simple way to 
>>> tie code to its dependencies.
>>
>> If this is all that people wanted, then we could just define 
>> repositories and associated metadata and forget about all runtime 
>> modularity support, but I strongly disagree that this is all people 
>> want or else there wouldn't be such an upsurge in OSGi interest and 
>> adoption.
>>
>>> I strongly suggest KISS applies to our effort.  I write this knowing 
>>> that the committee seems to be heavily in favor of this alternate 
>>> view that we implement the kitchen sink.
>>
>> I think our interest is similar, but perhaps our conclusions are 
>> different. Initially, I think my comments were construed as an 
>> argument against split packages in general, but I am really arguing 
>> the same as you (I think) that we don't really need split packages to 
>> be a concept in our "interoperability API", if that is how we are to 
>> view the 277 API. Of course, I am also willing to have the former 
>> argument too, but I will leave that for another time. :-)
>>
>> From my point of view, I do not think it is wholly reasonable to 
>> assume that we want to try to accomplish interoperability across 
>> module systems, where that includes arbitrarily sharing split 
>> packages across module systems.
> Nor do I, but I'm trying to understand the implications if we assume 
> that OSGi will continue to split packages.
>>
>> Let me try to accurately conjure the peculiarities of split packages 
>> in the OSGi module layer...
>>
>> We support split packages in two ways, via required bundles (i.e., 
>> module dependencies) and via bundle fragments. In general, we 
>> recommend that if you split your packages that you attach mandatory 
>> attributes on them so that importers do not get them by mistake. By 
>> requiring a bundle, you can ignore mandatory attributes and get 
>> everything a bundle has to offer and combine it with other required 
>> bundles that might be exporting other parts of the split packages. 
>> Given this, there is a semi-reasonable chance* that we can tell you 
>> which bundles are contributing to the "complete" packages, but we 
>> have no way of knowing if/when combined split packages are complete. 
>> Further, we have to assume that split package parts offered by 
>> different bundles do not overlap, because if we allowed them to 
>> overlap then we would have shadowing and ordering issues. Again we 
>> have no easy way to verify that they do not overlap, so we just 
>> assume they don't.
>>
>> Fragments on the other hand are a little trickier, since split 
>> packages from them are loaded from the same class loader to provide 
>> package private access (which is not afforded for split packages in 
>> the require bundle approach). The "host" bundle doesn't really know 
>> about fragments and it is possible for fragments to shadow or extend 
>> packages in the "host" bundle. Since fragments provide their split 
>> packages in an implicit way (i.e., they largely just become part of 
>> the host bundle's class path), there is no easy way to determine if a 
>> fragment is contributing to a given package or not. (* This feature 
>> of fragments also makes it so it is not really possible to know who 
>> is contributing classes to a split package in the require bundle case.)
> Ok, so first I assume it would make sense for an OSGi Repository to 
> simply never return a ModuleDefinition for a fragment.

I am not sure I understand your point. Host bundles would be returned 
and during the resolve process, fragments might be attached to that 
host, thus modifying its content in ways that we cannot predict.

> And the "you don't know what you've got till it's resolved" problem is 
> going to be present in any module system for which package name is not 
> a required component of a dependency expression. But as long as 
> re-export is not the default behavior (it isn't), doesn't this really 
> just boil down to the leakage problem? And this is addressed in 277 
> the same as it is in OSGi: you must declare such "leaks" as re-exports 
> ("uses"). Not perfect, but probably good enough.
>
> So I think we're left with the issue of a single search visiting 
> multiple loaders that can return the same package. This won't happen 
> in the fragment case, since the "split" gets mended at runtime, but it 
> will in the require-bundle case. And this will manifest as multiple 
> entries in the import list which export the same package.
>
> But even this isn't an issue, as long as dependency resolution relies 
> solely on declared exports and re-exports of a module, and not on what 
> is available by browsing the imports of that module.
>
> So I'm back to my original statement that we just need to acknowledge 
> that package duplication may exist in an import list.
>
> Or am I missing something here?

I am just not sure why we want to make the fact that some module systems 
support split packages a visible concept in our "interoperability API", 
when it would be difficult to get interoperability around split packages 
across module systems when we already have issues with split packages 
within a single module system. Especially, given my description above, 
since we would not easily know which modules were contributing to a 
split package in OSGi.

-> richard

>>
>> I think that is fairly complete description of the situation, sorry 
>> if it is somewhat terse. In summary, I wouldn't assume that we can 
>> get reasonable interoperability at this level. If you need these 
>> types of features, then you should stick to a single modularity 
>> framework...preferably OSGi. ;-)
>>
>> But my overall point is, I believe that if we can support 
>> interoperability at module- and package-level dependencies across 
>> module systems, then I think we would hit most use cases.
>>
>> -> richard
>>
>>>
>>> Sam
>>>
>>> On Apr 25, 2008, at 1:57 PM, Richard S. Hall wrote:
>>>
>>>> Bryan Atsatt wrote:
>>>>> Richard S. Hall wrote:
>>>>>> Gordon Hirsch wrote:
>>>>>>>> 1. Map its dependency declarations into a standard runtime 
>>>>>>>> representation.
>>>>>>>> 2. Support a standard search model over its stored modules, 
>>>>>>>> using the runtime dependency expressions.
>>>>>>>> 3. Map its stored module data into a standard runtime 
>>>>>>>> representation that can be returned by the search.
>>>>>>>
>>>>>>> One challenge lies in defining the "standard runtime 
>>>>>>> representations" and "standard search model" in a universal 
>>>>>>> enough way to encompass OSGi and other module systems. This 
>>>>>>> implies embracing concepts (in these standard representations 
>>>>>>> and search model) that were not universally liked by the EG 
>>>>>>> early on. (Split packages and package-level import/export come 
>>>>>>> to mind.)
>>>>>>
>>>>>> Package-level import/export seem like a must, but I still don't 
>>>>>> see split packages as a necessity.
>>>>> Strongly agreed on import-by-package, and that ModuleDefinition 
>>>>> requires a method to enumerate exported packages (and probably an 
>>>>> exportsPackage(String packageName) method to enable an efficient 
>>>>> Query).
>>>>> On the split package front, however, it seems a little fuzzy to 
>>>>> me. If an OSGi implementation of 277 is also going to remain OSGi 
>>>>> Rx compliant, it will still need to support split packages, right? 
>>>>> If so, don't we need to surface this fact at the 277 level? 
>>>>> Perhaps that is as simple as acknowledging that 
>>>>> Module.getImportedModules() may return more than one instance that 
>>>>> exports a given package.
>>>>
>>>> That all depends on if the 277 API is a subset of OSGi 
>>>> functionality or equal to it, I suppose.
>>>>
>>>> -> richard
>>>>>
>>>>> // Bryan
>>