Java extensibility and JPMS (ServiceLoader)

[Tomas Langer]

Helidon currently has around 300 modules with module-info.java. In general, this has improved our module structure and design.
Yet, we are now encountering some major issues related to extensibility.
 I will put down a few points that are problematic, and explain each in detail further in the e-mail (it is quite long, sorry about that).

1. provider implementations cannot be code generated without major problems
2. the provider interface module MUST be on module path, even if it could have `requires static`
3. the provider implementation must be public with public constructor
4. duality of definition between module path and class path

I am trying to propose a solution within the bounds of the current service loader design. Of course there may be other solutions (both with current design, or even creating a brand new extensibility solution in Java, this is just to illustrate it).

The first two issues are quite major, as they force us to recommend not to use JPMS to our users...

We could design our own extensibility approach, though that would require the use of reflection to instantiate the services, and we would have to live with the limits of the module system (where Java ServiceLoader works around a few of them).
I feel the right way is to use what Java provides, so I would welcome any help (and possibly changes in the language) to support our use cases.

Thanks,
Tomas Langer
Architect, project Helidon


Ad 1 - Code generation
-------------------------------------
Problem: We cannot code generate service implementations (well we can, but the user must handcraft them in module-info.java, so we end up running the APT, generating a service, failing the compiler to tell the user to add the service, compiling again every time a new service is added).
Possible solution: Provide extensibility to module-info.java that can be code generated
Without JPMS: it just works, as `META-INF/services` files can be code generated without issues

Details:
What I do not see is how we are supposed to do extensibility through annotation processing.
There are a lot of usecases for this, such as:
- generating code for serializers/deserializers for objects that persist to JSON, XML, YAML
- generating code for database entities
- generating descriptor for services in a service registry

We actually want to implement these three use cases, and it is a major pain for the user - I would not mind much if this hurt us, as framework developers, but we must force the user to take action by breaking the compilation, or come up with some really weird solution (such as source code modification using some preprocessor before compilation, or postprocessor running on bytecode to re-generate module-info.class)


Ad 2 - Provider module cannot be optional dependency
-------------------------------------
Problem: We cannot declare `requires static` on a module that has the ServiceLoader provider interface (or abstract class)
Possible solution: Change the rules for JPMS to allow this
Without JPMS: it just works, as `META-INF/services` to not impose any classpath structure

Details:
If a module (my.json) defines a provider interface (let's say `JsonSerializer`), and I create a module with `MyJsonSerializer` (provides JsonSerializer witih MyJsonSerializer), currently I MUST do the following "requires my.json". If I do a "requires static my.json" I fail to start the JVM if that module is not on module path.

The service CANNOT be used unless the module is on module path (as anybody attempting to load it must declare `uses` in their module info with a proper `requires` on the `my.json` module.

So what are my options right now?
- have a "requires my.json" and just dump the module on all my users (not so good - people may want to use my library without JSON altogether, I may also provide support for XML, YAML - all of these would need to be on module path)
- create a module for each (one for JSON, XML, YAML + my library) - resulting in 4 modules (and this may grow if I decide to support other format); this looks kind of OK on the first look, but with the number of modules we have, and the number of features we support, this gets out of hand really really quickly; this approach is also very user unfriendly, as now the user needs to understand 4 modules instead of just 1, and use the right ones at the right time).
- considering the number of modules we already have, this would make our project unmaintainable (and unusable for users)

As JPMS already allows "static" dependencies, there should be no reason not to allow it in this case as well. We can break the module system even now - just use a class from static dependency in a public class - this will fail at runtime only. The service loader is not different (this is a reaction to text in https://bugs.openjdk.org/browse/JDK-8299504).


Ad 3 - Provider implementation must be public with public constructor
-------------------------------------
Problem: This creates a new public API that we may not want to expose, or document
Possible solution: Change the rules to allow package local service provider implementations with package local constructors
Without JPMS: Same issue, even more problematic as there is no restrictions on package visibility

Details:
Provider implementations are not supposed to be visible to users - they are not public API of my module (the fact that I provide a service is part of my public API).
Right now there is only one option to work around this, and it only works in JPMS, and in my opinion it brings in even more problems - put the provider implementation in an un-exported package.
The problem with this approach is that now the provider implementation MUST use only public methods of my module, thus creating even more public APIs, where if I just put it in my exported package, I can use package private methods of my other classes to implement the service (so I pay the price of having one public class with one public constructor agains multiple public classes and public methods). Also the "hiding" in unexported package is lost when on classpath anyway...


Ad 4 - Duality of definition between classpath and module path
-------------------------------------
Problem: To support services, we MUST declare them twice - once in `provides` in module-info.java, once through META-INF/services
Possible solution: Java could read module descriptors even when running in classpath mode to add service implementations and merge it with META-INF/services information
Without JPMS: it just works, as `META-INF/services` is always honored on classpath and for non-JPMS modules on module path

Details:
This is again quite a pain for us as framework develoepers, and a pitfall for users. When we started with JPMS, we had both created manually, which obviously ended in a huge inconsistent mess.
So now we have a custom Maven plugin, that creates META-INF/services files based on the content in module-info.java and fails on inconsistencies.
I do not consider this a nice solution for us, and definitely not for end users. Also there is no way to find out that you forgot to add one (or the other), as JVM just does not care. So basically you end up with a runtime issue that is really hard to troubleshoot.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jigsaw-dev/attachments/20240119/0791b794/attachment-0001.htm>