Module declarations

[David M. Lloyd]

On 09/17/2015 09:52 AM, mark.reinhold at oracle.com wrote:
> (splitting your initial reply into separate threads)
>
> 2015/9/9 1:48 -0700, david.lloyd at redhat.com:
>> On 09/09/2015 11:51 AM, mark.reinhold at oracle.com wrote:
>>> To get the discussion started I suggest we focus first on the high-level
>>> aspects of the design covered in "The State of the Module System" [1].
>>>
>>> [1] http://openjdk.java.net/projects/jigsaw/spec/sotms/
>>
>> 1) The first thing that jumps out at me (again) is making module
>> declarations be a binary construct produced by the actual programming
>> language.  It strikes me as arbitrary as well as redundant (given
>> there's a programmatic API to build descriptors, they could simply be
>> parsed from text nearly as well, and a lot more nicely from the user's
>> perspective).  As pointed out in the document, it is already expected
>> that additional tooling would synthesize this file at build time,
>> further decreasing the value of such an idea.  The module class file
>> just seems pointless to me.
>
> One of our requirements is fidelity across all phases.  To achieve this,
> javac (or whatever Java compiler you use) must be able to locate and
> interpret descriptions of modules.  If modules are not described by
> language constructs that are compiled into class files, then how and
> where would you suggest that a compiler find such information?

They could still be bundled with the artifact, but in clear text format 
perhaps.  Or even in binary format, but in either case, the descriptor 
format should be specific to a specific default module configuration 
(e.g. an application module loader) rather than be a hard-wired aspect 
of all modules.

To me "fidelity" means the same thing as "integrity": I can build an 
artifact and expect it to run on any environment which meets its ABI 
requirements in any way.  I don't think that hard-coding module 
dependency names in the artifact itself is a good way to achieve this 
though.  Doing so hampers the ability to separate building from 
packaging, since if you need to specify different run time environment 
parameters in the packaged version, you must crack open and modify the 
artifact to do so (new SHA1/MD5 checksums, etc.).

To me a better way to achieve this is a multi-tiered approach, where 
each tier depends on the tier before (but not vice-versa).

The build artifact is identified and versioned as today (Maven-style), 
and packaged alongside associated build dependency information.  This 
information is global - i.e. it is the same regardless of where and how 
the artifact is used, and the artifact implicitly has a clear ABI. 
Building a new artifact only depends on build dependency information 
associated with the dependency artifacts.  This method of building is 
well-proven.

The next tier is a packaging/distribution tier.  This pertains 
specifically to the environment in which artifacts are utilized.  In 
this tier, modules are defined which consist of the combination of build 
artifacts (with versions) and run-time, distribution-specific dependency 
information.  The distribution could (and should) be versioned as a 
whole; individual module versions within the environment may correspond 
with the artifact(s) that comprise it (but this is not necessary).  The 
distribution information, being versioned, could easily be distributed 
wholly or in sections on a system like Maven.

An important aspect of this tier is testing, and its role in managing a 
stable module distribution.  When a new version of an artifact is 
produced and made generally available, before a module may be included 
in a distribution it is necessary to determine whether such inclusion 
will regress any aspect of the existing infrastructure.

While completely retesting an entire distribution every time a component 
or combination of components is updated is generally infeasible, it is 
possible to use the distribution's run-time dependency information to 
attain a level of confidence regarding the impact of an upgrade, by 
working downwards along the dependent graph incrementally from each 
changed module until the desired confidence is achieved that consumers 
(direct or indirect) are unlikely to be regressed.

The last tier is the run time tier, in which some or all of the module 
graph of a distribution is made available for execution, presumably (but 
not necessarily) local to the running JVM.  No version information is 
used at run time outside of an informational capacity.

The reasons that this system is superior to a "flat" system are 
hopefully obvious: what you run is not the same as what you build, and 
no system exists which can make it be the same, unless you introduce a 
central broker to all future Java builds which enforces consistency.  A 
"version" of a module is really a product of the version of the artifact 
and the environment in which the module is executed.  It is highly 
unlikely that restricting usage of the module system to a single 
effective distribution will suffice for most, or even many, use cases. 
But this is exactly what will effectively happen by flattening run time 
information all the way down to the build time layer: the result will be 
one massive, headless module ecosystem whose overall security and 
stability is completely indeterminate, which evolves in a completely 
unpredictable manner.  This is the complete opposite of fidelity in my 
mind, and solves nothing of practical value.

> (The statement that "build tools, in particular, can synthesize module
>   declarations from information already available in project descriptions"
>   in the SotMS document is, I see, overly terse.  A tool could suggest an
>   initial module declaration with `requires` clauses derived from a
>   pom.xml file, or equivalent, but it'd be up to the developer to specify
>   the rest.  I'll fix this in the next version.)

Yes I agree that a build tool can, at best, only make suggestions based 
on information it may have about the target environment (which is 
conceptually not very much unless the build tool becomes highly aware of 
packaging considerations).  Only something like a packaging tool that 
has awareness of the distribution as a whole can give really useful 
dependency hints for run time though.
-- 
- DML