Inconsistency with service loading by layer or by class loader

[David Lloyd]

I'm once again experimenting with modularizing some of our runtime projects
(Quarkus, WildFly) using JDK modules. Among the various problems
I've encountered, this one is one I haven't cracked yet and seems like an
oversight.

I am currently experimenting with a one-module-per-layer design which
allows us certain capabilities (like lazy loading of modules, late binding
of dependencies, and circularity in dependencies), and also allows us to
create module graphs that can mix in "unnamed" modules for libraries which
don't yet work properly as named modules, as well as automatic modules for
libraries which can work as modules but don't yet define a descriptor.

However I have discovered an important difference between service loaders
which load services by class loader (for example, those found in
Microprofile and Jakarta frameworks) and those which load by module layer
(none?) which is preventing things from working properly.

When I load a service by layer, after searching the given layer, the parent
layers of that layer are then searched in order, recursing up the graph.
(Sometimes the same service could be returned multiple times when there are
diamonds in the layer graph, but that's a different problem.)

When I load a service by class loader, if the service is not found in the
class loader's defined layers, then the search terminates and the service
is considered not found, even if the service provider exists in a parent
layer of one of those layers. This essentially means that all frameworks
which are loading services by class loader (and using the class loader of
the framework itself is typical here) will also need all of their
implementations to coexist in one of the layers defined by that class
loader - or else they must be defined in the unnamed module (in which case
it will always be found).

Ironically, it *will* search the parent class loading delegation chain for
layers which contain service implementations. The point of using module
layers though is that they can have multiple parents, departing from this
single-delegation model. In our class-loader-per-module setup, we do not
use the parent class loader delegation as it would be meaningless in that
context.

A good solution would be to modify service loading by class loader to also
search parent layers of each layer defined to that class loader. This seems
like it would be fairly easy to implement, but might possibly have subtle
compatibility implications for very specific environments which have a
module-per-class-loader situation along with multiple layers.

Failing that, the best solution would be to allow containers which do
dynamic loading to update the service catalog of the class loaders and/or
layers that they control as needed. We already work around the lack of
`Controller.addUses` by defining a class in each module that we control
which gives us a `MethodHandle` to do this on their own behalf (which, btw,
is a poor solution but it works). We could just as well use reflection to
crack open the `addProvider` method from `jdk.internal`, and this would
solve our problem simply and somewhat cleanly, give or take compatibility
guarantees. Better though would be to add this method to `Controller` and
let user layers just be dynamic/late-binding with service registration (and
usage) - they are essentially already dynamic in almost every other
respect. It seems clear after 7 years of hindsight that nothing of
practical relevance would be lost by such a solution (and I would be happy
to manage the bug, CSR, and PR for this if it was likely to be accepted).

Another possible solution would be to work around the problem in-place: for
example, defining a "bogus" module in the service provider's module layer
and defining that module in the service user's class loader so that the
provider's layer appears in the user's class loader. This is a poor and
expensive solution.

Another solution is to locate all frameworks which are loading services by
class loader, and have them load by layer as well, merging the results.
This solution is prone to complexities though (service order sometimes is
significant, duplicates may come through) and working around these problems
will likely result in inconsistent implementations between frameworks.
Also, it is unlikely that it would be possible to contribute appropriate
changes to *all* frameworks using this pattern with a high degree of
confidence that they would be merged and promptly released/backported.

Another solution would be to add in a "provides" directive to every module
which uses a service, where the "provides" directive gives a list of class
names for each service type like `$internal.ServiceProxy1_ServiceType`,
`$internal.ServiceProxy2_ServiceType`, etc. up to some number. Then the
class loader can dynamically generate service providers which then use a
separate registry to find the service implementation, returning it via the
`provides` method of each generated class. This is a poor & expensive
solution as well, and also falls short in the case where a module
dynamically adds a `uses` for whatever reason, which it is allowed to do at
any time.

-- 
- DML • he/him
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jigsaw-dev/attachments/20241211/722bead0/attachment.htm>