Selectively Shifting and Constraining Computation

[Mark Reinhold]

2022/11/7 7:57:50 -0800, Dan Heidinga <heidinga at redhat.com>:
> On Thu, Oct 27, 2022 at 8:28 PM mark.reinhold at oracle.com wrote:
>> 2022/10/21 19:49:36 -0700, Dan Heidinga <heidinga at redhat.com>:
>>> Thanks for posting this document.  It provides a clear starting point for
>>> us to move forward to start to address Leyden.
>> 
>> Thanks, Dan, and thanks for your comments and questions.
>> 
>>> A few questions based on my read through:
>>>
>>>> As long as a condenser preserves program meaning and does not impose
>>>> constraints other than those accepted by the developer, an implementation
>>>> has wide latitude to optimize the result.
>>>
>>> Does “an implementation” refer only to the JDK or are other parts of the
>>> ecosystem - frameworks, libraries, etc - able to add their own condensers
>>> as well?
>> 
>> Good question!
>> 
>> We’re concerned here primarily with any software system that’s claimed
>> to be a conformant implementation of the Java Platform -- the JDK is
>> merely the dominant and best-known example.  The components of such a
>> system -- including any condensers -- can be held to the rigors of the
>> Platform Specification and the JCK, thereby ensuring that they preserve
>> the meanings of Java programs.
>> 
>> Some people will, no doubt, want to write their own condensers, and
>> obviously we’d like all condensers to preserve program meaning.  The
>> current conformance regime (specification/RI/JCK), however, applies only
>> to entire Platform implementations.
> 
> Interesting.  I'm hesitating to raise this as I don't want to derail the
> technical topics, but I recall that after jlink was released, it was
> necessary to update the definition of the "platform" to handle the subset
> runtimes created by jlink.

We updated the definition of the Platform Implementation concept to
allow conforming module subsets [1] prior to releasing jlink in JDK 9,
not after the release.

>                             With the kinds of changes envisioned for
> condensers - including removing unused classes, fields & methods - will we
> need to update the definition of the platform again?

Almost certainly.

>                                                       Updates of this
> nature tend to take a while on the legal front so I'm raising this early
> (and hoping it's already being thought about) so the project doesn't end up
> with conderners that can't be shipped due to conflicts with the platform
> definition.

We’ll evolve the Platform Specification to admit condensers, and
whatever opt-in constraints they might impose, prior to shipping any
actual condensers.

As usual, moreover, we’ll evolve the Specification in parallel with the
implementation, so that the work on each is informed by the work on the
other.  (I know I don’t need to explain that to you, but some people
assume that these are non-concurrent efforts, with one driving the
other.)

>>>> ...
>>>> 
>>>> Condensation is meaning preserving.
>>> 
>>> Have you given thought to precisely specifying and then validating this
>>> property?  It’s definitely something we want but it may be challenging to
>>> enforce….
>> 
>> The meaning of a Java program is already given by the Specification, and
>> the JCK already validates a Java implementation by running a large
>> collection of Specification-derived tests.  My initial thought here is
>> to extend the JCK to test a condenser by applying the condenser to each
>> JCK test and then verifying that the condensed test still passes.  (To
>> keep JCK run times tractable, do this in large batches.)
> 
> That's a great starting point.  I hadn't considered using the JCKs in that
> way.
> 
> The devil will be in the details as there will be conflicts between the
> condenser's changes (make this module unmodifiable) and the test's
> expectations (redefine class X).

Yes.  Many existing tests will likely need adjustment.

>                                   This approach also feels a bit weighted
> toward the end - a condenser can be developed, merged, and only later be
> discovered to conflict with the JCKs.  I've been trying to think of ways to
> bring this forward in the development process and to make it more open but
> the answer may be as it is today - lots of jtregs, clearly written specs,
> and the final check is done by the JCKs.

Agreed.

>>>> ...
>>>> 
>>>> Eliminate apparently-unused classes and class members (i.e., stripping, which
>>>> requires the constraint that selected classes and members cannot be reflected
>>>> upon)
>>>
>>> Being slightly pedantic, it means the removed members can’t be reflected
>>> on, correct?
>> 
>> Yes, but ...
>> 
>>>               A class that’s been stripped of unused methods / fields can
>>> still have its remaining members reflectively looked up is my assumption.
>> 
>> That’s a reasonable assumption.
>> 
>> In detail, though, we could in principle retain reflective metadata even
>> for removed program elements, if there’s a need to do so.  We could,
>> likewise, remove reflective metadata for some retained program elements
>> if we’re confident that those elements will never be reflected upon.
>> More exploration required ...
> 
> That makes sense.  If I'm following that correctly, we can mark some
> classes as "reflection hostile" and others as supporting reflection of all
> members (even the removed ones).  What I'm really hearing is a willingness
> to completely change the model so users can make different tradeoffs.

Exactly.

>>> Using jlink is the natural starting point for this work.  It would be
>>> unfortunate if condensers were limited to only work on modules given the
>>> (unfortunately) slow update of them.  Is the intention to extend jlink to
>>> work with classpath entities (ie: non-modular code) as well?  We foresee
>>> the need for condensers to be able to “condense” non-modules as well.
>> 
>> The jlink tool doesn’t support arbitrary class path code, or automatic
>> modules, in order to guarantee that the resulting image is structurally
>> correct by construction.  I’m reluctant to give up on that guarantee.
>> Viable workarounds exist for non-modular code (jdeps, ModiTect, etc.),
>> so developers who want to risk building a potentially-broken image from
>> a potentially-broken class path can do so, at their option.
> 
> Libraries are always the challenge here. Modularizing before your
> dependencies is difficult and while Leyden may help drive modularization in
> the ecosystem (a plus!), I hope we can take a pragmatic approach and bring
> class path code into Leyden fold.  As you said above, lot's to explore here
> but I ask that we add an explicit goal to bring class path code in as much
> as we can.

If all condensers must support the class path then none will be able to
leverage the constraints imposed by modules -- and as I said, I suspect
that there are some interesting ways to leverage those constraints.

It might make sense to extend jlink to copy the entire content of the
class path (modulo duplicates) into a custom run-time image, but I don’t
think we should require every condenser to support the class path.

- Mark


[1] https://cr.openjdk.java.net/~iris/se/9/java-se-9-fr-spec/#Constraints-on-all-modules-in-an-Implementation