[External] : Re: Inconsistency with service loading by layer or by class loader

[David Lloyd]

On Mon, Dec 16, 2024 at 11:56 AM Ron Pressler <ron.pressler at oracle.com>
wrote:

>
>
> > On 16 Dec 2024, at 17:17, David Lloyd <david.lloyd at redhat.com> wrote:
> >
> > The model I'm working with is such that we are loading every single
> module (whether they are within our container or the user's application)
> into a separate layer, as has been previously recommended as the way to
> meet requirements like ours, with our class loaders handling all
> inter-module linkage concerns. Everything works exactly as we want in this
> scenario: modules are able to be lazily loaded and linked (just like
> classes); our integrity constraints are enforced locally per module on
> demand, rather than globally across the whole layer (again, just like
> classes); and, we get our nice stack traces and encapsulation behaviors.
> However there is one exception, and that is that service loading does not
> work for any module, since no modules can find their implementations,
> regardless of what we do with class loading. This is what I'm trying to
> address.
>
> Ok, thank you! Indeed, if a dynamic container is designed to load every
> module in a separate layer then ServiceLoaders in libraries may not work
> unless the container is able to put a library’s service providers in the
> same layer as the library.
>
> Now, if you’re not interested in Alan’s suggestion, then there’s not much
> I can add.
>
> I can say that I’ve seen similar problems arise with virtual threads. Some
> projects wanted to get the benefit of virtual threads under the condition
> that nothing else in their architecture, algorithms, or deployment would
> have to change, and that’s just not possible sometimes (e.g. to get the
> benefit of virtual threads you need high concurrency, and if the rest of
> your system is designed and deployed to ensure that every server only ever
> experiences low concurrency then you won’t enjoy the benefits of virtual
> threads). Similarly, modules were not designed to match every possible
> non-modular design (e.g. they also forbid circular dependencies),
> especially every elaborate dynamic loading design. That wasn’t the intent.


> As another side note, I will also say that not a single invariant written
> in Java can have integrity without modules’ strong encapsulation. Even
> properties such as “strings are immutable” cannot be trusted, let alone
> “this private method is never called unless some precondition holds”. In
> particular, class loader isolation only offers encapsulation in a very
> approximate and rather weak sense.
>

If you look back at the old jigsaw-dev archives, you'll see my name, and
those of my colleagues, quite frequently. I am well aware of the intent of
this system and how it was implemented. We do not (and you should not)
consider these specifications and their implementations to be holy writ.
They are in fact no different from those of any other project, specifically
that they can and should be changed as needed. I am expressing such a need.

Also, you should be careful with words like "intent". In the specification
of a computing system, "intent" effectively counts for nothing. That, more
than anything, is what makes the JVM a successful application platform: the
specification is detailed and complete, and what is allowed or disallowed
is governed only by the specification (not by any nebulous concepts like
"intent" or "principle"). Without such a tight specification, compatibility
would be impossible, as would interoperability and portability.

In short, I’m not discounting the difficulty you’ve encountered, but if you
> want to enjoy modules’ capabilities as they currently work, you may want to
> consider a different design.


I'm asking for the OpenJDK team to instead consider a different design for
service loading in the Java platform itself, in the absence of an
alternative solution which can use what exists today. Unfortunately, the
constraints around the JPMS are such that I do not believe there is another
design that will generally work for us. I do believe that there are
possible workarounds which belie the conceit that the encapsulation of the
module system is as strong as you present it; the existence of
`ModuleLayer.Controller` itself shows that it is nowhere near absolute.
This level of encapsulation is much more superficial than, say, member
access control, which is to say that it can be worked around in various
ways without "breaking the rules" (for example, by having a class loader
define generated helper classes in every package of a module, I can gain
access to their original Lookup objects, breaking *any* form modular of
encapsulation without breaking any of the rules of the platform
specification). If you would recoil at such a thing, consider again the
difference between "intent" and "specification".

I can tell you that several design choices of the JPMS had nothing to do
with encapsulation principles and everything to do with the philosophy of
those who created the constraints. There's nothing inherent in the system
that makes it necessary to prevent circularity, *or* to eagerly load all
modules in a layer, *or* to eagerly resolve services. I'm not saying this
to beat a (very) dead horse, but because it's important to understand that
these were not *necessary* design choices, they were based on opinionation.
Opinionated systems are not necessarily "bad", but ultimately the reason
these design decisions were accepted were not due to consensus but due to
politics. We've accepted this and have moved on, however there are real
implications to these choices that mean that many kinds of Java application
containers cannot be satisfactorily migrated to use JPMS modules. I hope
you recognize that I am trying to change that, but that doing so may
require some form of compromise from the platform itself. The
`ModuleLayer.Controller` API demonstrates that such compromise is possible.

Is Quarkus also experiencing that difficulty?
>

Yes. Quarkus currently does not use JPMS modules for the user application
at build time or in any deployment mode, both of which instead rely on an
arrangement of specialized class loaders, with all classes ending up in
flat unnamed modules. My current experiments revolve around trying to
create more modular-oriented and encapsulated packaging options.

-- 
- DML • he/him
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jigsaw-dev/attachments/20241216/c55ead17/attachment-0001.htm>