Java extensibility and JPMS (ServiceLoader)

[Josiah Noel]

I help maintain a few service-loader-based modular annotation processor
libraries with the avaje framework, so I have also run into this and had to
work around it.

*Ad 1:*
So this is indeed an inconvenience but in my experience, it's not a crazy
one. When processing is over we check the module-info's directives to throw
compilation warnings like:
*[ERROR]
/M:/Dev/avaje-helidon-nima-api-example/src/main/java/module-info.java:[1,1]
Missing `provides io.ava*
*je.http.client.HttpClient.GeneratedComponent with
com.jojo.helidon.api.client.httpclient.GeneratedHttpCompon*
*ent;`*
*[ERROR]
/M:/Dev/avaje-helidon-nima-api-example/src/main/java/module-info.java:[1,1]
Missing `provides io.ava*
*je.jsonb.Jsonb.GeneratedComponent with
com.jojo.helidon.api.jsonb.GeneratedJsonComponent;`*
*[ERROR]
/M:/Dev/avaje-helidon-nima-api-example/src/main/java/module-info.java:[1,1]
Missing `provides io.ava*
*je.validation.Validator.GeneratedComponent with
com.jojo.helidon.api.controller.valid.GeneratedValidatorComp*
*onent;`*
*[ERROR]
/M:/Dev/avaje-helidon-nima-api-example/src/main/java/module-info.java:[1,1]
Missing "provides io.ava*
*je.inject.spi.Module with com.jojo.helidon.api.ApiModule;"*

The *real problem *with checking the module info in a processor is that
ModuleElement is bugged such that if you call *ModuleElement.getDirectives*
on the project module It breaks compilation
<https://bugs.openjdk.org/browse/JDK-8315125>. (they called this one a
duplicate but I still have the problem even in JDK 22-ea and JDK 23-ea)

To get around this I have to use Filer to retrieve the module-info's
sources file and parse it as a string to avoid calling *getDirectives. * I've
built libraries to do this to make this easier, but the problem remains
that if any other processor uses *getDirectives*, compilation will still
break.

*Ad 2:*
I asked this question here a while back, and what I got out of it is that
we needed to do some form of circular dependency if we truly wanted
optional services, the tech for compiling multi-module jars isn't here yet.
It's a pain to deploy when we change the plugins, but we rarely do so this
has worked fine for us. Example: avaje-validator
<https://github.com/avaje/avaje-validator> provides a plugin for avaje
inject that itself depends on avaje-validator.

*Ad 3: *
Yeah I got nothing, we never had to do this.

*Ad 4: *
This one was the simplest, we can define an annotation to go on the user's
service class and process them to generate META-INF files and validate the
module-info. In this way, one cannot forget to add the proper module-info
information. In some of the libs we do this with their processors, but we
also have a dedicated library for handling this sort of thing.


On Fri, Jan 19, 2024 at 8:04 AM Tomas Langer <tomas.langer at oracle.com>
wrote:

> Helidon currently has around 300 modules with module-info.java. In
> general, this has improved our module structure and design.
> Yet, we are now encountering some major issues related to extensibility.
>  I will put down a few points that are problematic, and explain each in
> detail further in the e-mail (it is quite long, sorry about that).
>
> 1. provider implementations cannot be code generated without major problems
> 2. the provider interface module MUST be on module path, even if it could
> have `requires static`
> 3. the provider implementation must be public with public constructor
> 4. duality of definition between module path and class path
>
> I am trying to propose a solution within the bounds of the current service
> loader design. Of course there may be other solutions (both with current
> design, or even creating a brand new extensibility solution in Java, this
> is just to illustrate it).
>
> The first two issues are quite major, as they force us to recommend not to
> use JPMS to our users...
>
> We could design our own extensibility approach, though that would require
> the use of reflection to instantiate the services, and we would have to
> live with the limits of the module system (where Java ServiceLoader works
> around a few of them).
> I feel the right way is to use what Java provides, so I would welcome any
> help (and possibly changes in the language) to support our use cases.
>
> Thanks,
> Tomas Langer
> Architect, project Helidon
>
>
> *Ad 1 - Code generation*
> -------------------------------------
> Problem: We cannot code generate service implementations (well we can, but
> the user must handcraft them in module-info.java, so we end up running the
> APT, generating a service, failing the compiler to tell the user to add the
> service, compiling again every time a new service is added).
> Possible solution: Provide extensibility to module-info.java that can be
> code generated
> Without JPMS: it just works, as `META-INF/services` files can be code
> generated without issues
>
> Details:
> What I do not see is how we are supposed to do extensibility through
> annotation processing.
> There are a lot of usecases for this, such as:
> - generating code for serializers/deserializers for objects that persist
> to JSON, XML, YAML
> - generating code for database entities
> - generating descriptor for services in a service registry
>
> We actually want to implement these three use cases, and it is a major
> pain for the user - I would not mind much if this hurt us, as framework
> developers, but we must force the user to take action by breaking the
> compilation, or come up with some really weird solution (such as source
> code modification using some preprocessor before compilation, or
> postprocessor running on bytecode to re-generate module-info.class)
>
>
> *Ad 2 - Provider module cannot be optional dependency*
> -------------------------------------
> Problem: We cannot declare `requires static` on a module that has the
> ServiceLoader provider interface (or abstract class)
> Possible solution: Change the rules for JPMS to allow this
> Without JPMS: it just works, as `META-INF/services` to not impose any
> classpath structure
>
> Details:
> If a module (my.json) defines a provider interface (let's say
> `JsonSerializer`), and I create a module with `MyJsonSerializer` (provides
> JsonSerializer witih MyJsonSerializer), currently I MUST do the following
> "requires my.json". If I do a "requires static my.json" I fail to start the
> JVM if that module is not on module path.
>
> The service CANNOT be used unless the module is on module path (as anybody
> attempting to load it must declare `uses` in their module info with a
> proper `requires` on the `my.json` module.
>
> So what are my options right now?
> - have a "requires my.json" and just dump the module on all my users (not
> so good - people may want to use my library without JSON altogether, I may
> also provide support for XML, YAML - all of these would need to be on
> module path)
> - create a module for each (one for JSON, XML, YAML + my library) -
> resulting in 4 modules (and this may grow if I decide to support other
> format); this looks kind of OK on the first look, but with the number of
> modules we have, and the number of features we support, this gets out of
> hand really really quickly; this approach is also very user unfriendly, as
> now the user needs to understand 4 modules instead of just 1, and use the
> right ones at the right time).
> - considering the number of modules we already have, this would make our
> project unmaintainable (and unusable for users)
>
> As JPMS already allows "static" dependencies, there should be no reason
> not to allow it in this case as well. We can break the module system even
> now - just use a class from static dependency in a public class - this will
> fail at runtime only. The service loader is not different (this is a
> reaction to text in https://bugs.openjdk.org/browse/JDK-8299504).
>
>
> *Ad 3 - Provider implementation must be public with public constructor*
> -------------------------------------
> Problem: This creates a new public API that we may not want to expose, or
> document
> Possible solution: Change the rules to allow package local service
> provider implementations with package local constructors
> Without JPMS: Same issue, even more problematic as there is no
> restrictions on package visibility
>
> Details:
> Provider implementations are not supposed to be visible to users - they
> are not public API of my module (the fact that I provide a service is part
> of my public API).
> Right now there is only one option to work around this, and it only works
> in JPMS, and in my opinion it brings in even more problems - put the
> provider implementation in an un-exported package.
> The problem with this approach is that now the provider implementation
> MUST use only public methods of my module, thus creating even more public
> APIs, where if I just put it in my exported package, I can use package
> private methods of my other classes to implement the service (so I pay the
> price of having one public class with one public constructor agains
> multiple public classes and public methods). Also the "hiding" in
> unexported package is lost when on classpath anyway...
>
>
> *Ad 4 - Duality of definition between classpath and module path*
> -------------------------------------
> Problem: To support services, we MUST declare them twice - once in
> `provides` in module-info.java, once through META-INF/services
> Possible solution: Java could read module descriptors even when running in
> classpath mode to add service implementations and merge it with
> META-INF/services information
> Without JPMS: it just works, as `META-INF/services` is always honored on
> classpath and for non-JPMS modules on module path
>
> Details:
> This is again quite a pain for us as framework develoepers, and a pitfall
> for users. When we started with JPMS, we had both created manually, which
> obviously ended in a huge inconsistent mess.
> So now we have a custom Maven plugin, that creates META-INF/services files
> based on the content in module-info.java and fails on inconsistencies.
> I do not consider this a nice solution for us, and definitely not for end
> users. Also there is no way to find out that you forgot to add one (or the
> other), as JVM just does not care. So basically you end up with a runtime
> issue that is really hard to troubleshoot.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jigsaw-dev/attachments/20240119/3b667438/attachment-0001.htm>