Module-system requirements

[mark.reinhold at oracle.com]

2015/2/13 6:06 -0800, tim_ellison at uk.ibm.com:
> Overall comments:
> 
>  - I have quoted sections below in providing feedback.  Hopefully that 
> works well enough for you.  You may want to consider sharing the 
> markdown/odf document if you want comments and questions in the full 
> context.

Good idea -- publishing the Markdown source (not odf, ugh) would be
straightforward.  I'll look into it.

> - The requirements doc is a mixture of specification and implementation 
> requirements.  For example, while I agree with the objectives of the 
> linking and packaging sections, they could be more succinctly written for 
> their requirements on the module system itself.

Yes, some of the requirements are implementation-oriented but, as such,
they do imply requirements upon the specification.  I'm not enough of a
requirements purist to be bothered by that, but I'm open to adopting
more-abstract rewrites of these items if you'd like to draft them.

>  - It would be useful to have a section on Security.  While there are 
> aspects throughout other sections of the requirements, such as 
> encapsulation and referential integrity, there may be other considerations 
> where module boundaries can be used to enhance security, e.g. only 
> allowing privilege escalation within a module scope, modules declaring all 
> exported APIs fail when passed a tainted string, etc.  I'm not proposing 
> scope-creep but rather ensuring folks have had the chance to think about 
> it and easy opportunities are not missed to make Java 9 even more secure.

I'm all for improving security (obviously!), but if I've learned one
thing about that general topic over the years it's that it's more a
state of mind, or a way of life, than a set of features.  It runs
through everything that we do.

Putting every requirement that's somehow related to security into a
special "security" section of this document would just make it harder to
read, and I'm skeptical that it'd actually make us all think even more
about security than we already do.  If you'd like to propose additional
security-related requirements then by all means please go ahead, and
we'll see where they fit into the present structure.

>  - I wrote my comments before reading other people's.  Apologies to David, 
> who did a better job of explaining the 'Exporting things' issues.
> 
> Detailed comments follow:
> 
>> Fundamentals
>> 
>> Encapsulation — ...
> 
> The document explicitly calls out classes and interfaces here as targets 
> for encapsulation.  Given a module can also contain native code, static 
> resource files, and user-editable configuration files, are these being 
> explicitly excluded?  It would be valuable for the module system to also 
> prevent access to libraries and resources that are not exported by their 
> containing modules.
> 
> I appreciate this requires extending the Exports to include non-package 
> resources, but the alternative is that native code and resources that are 
> specific to a module's implementation would be accessible to other modules 
> (if the default were to export resources), or that modules cannot access 
> resources managed by other modules (if the default were to not export 
> resources).

As I wrote in my reply to David, I think resources (in the traditional
ClassLoader::getResource sense) should be treated as being strictly
internal to modules.  If a module needs to make the information in one
of its resources available to other modules then it should do so via a
properly-exported class, or a service provider.  Just because JAR files
are miniature filesystems which anybody can dissect doesn't mean that we
need to conceive of Java modules in the same way.

As to the other categories of information in a module:

  - I'm not sure it's worth going to great lengths to prevent a native
    library in one module from being accessed by another module.  If a
    module has the power to load or access native code then it can
    pretty much do anything it wants.  Taking stronger measures here
    would bring low benefit for high cost.

  - Configuration files are a pretty special case.  They're meant for
    things like property files which need to wind up in the final
    run-time image and might be edited by a deployer or end user.  They
    are, in some sense, implicitly exported to all other code, though
    the only way to access them is by knowing their final location in
    the file system relative to ${java.home} (arguably there should be
    an API for that).  I don't think it's worth trying to invent a
    mechanism to prevent one module from reading another's configuration
    files.

(Confession: In some ways I'm a bit queasy about including configuration
 files in this specification at all, but they're critical to the
 modularization of both the SE Platform and its RI, the JDK.)

>> Non-interference — The Java compiler, virtual machine, and run-time
>> system must ensure that modules that contain packages of the same
>> name do not interfere with each other.
> 
> Can you explain "interfere"?  Is this referring to ambiguous class and 
> linkage error type problems, or something else here?

The former.  If two distinct modules contain packages of the same name
then code in that package in one module must not be able to access
package-private types or members in that package in the other module.
If two distinct modules contain classes of the same name (i.e., with the
same binary name) then one must not be able to stand in for the other.

>> Compatible Java Platform modularization — It must be possible to
>> divide an existing Java Platform, e.g., Java SE or Java EE, into a
>> set of modules such that existing libraries and applications can run
>> without change, so long as they use only standard platform APIs.
> 
> It is the last nine words of this that I have concerns about.  With that 
> caveat I think there is a serious risk that Java 9 will be incapable of 
> running a large body of existing, valuable Java code.  There is a longer 
> discussion to be had about the support for 'legacy' Java code, but just as 
> the document provides for 'gradual migration of applications' under 
> Development, I believe there is value in a promiscuous mode for the 
> modular platform or specific code source in being able to run existing 
> libraries and applications without change.

This is an important point, but remember that the requirements we're
discussing here are upon the module system itself rather than upon the
modularization of the Java SE Platform which it will enable.  The latter
will be a topic for the Java SE 9 EG, and I'm sure we'll have vigorous
discussions there about the extent to which legacy code that depends
upon internal interfaces should continue to work.

Having said that, I do agree that the module system must provide some
kind of escape hatch or "promiscuous mode", as you call it, so here's
a stab at an additional requirement for the "Modularizing Java SE"
section:

  - _Overrideable encapsulation_ --- It must be possible to force a
    module to export one or more of its packages to all other modules,
    or to some specific modules, even if its definition does not do so.

>> Upgradeable modules — If a Java SE module exports an endorsed
>> standard or standalone technology then, given a particular
>> implementation of the platform, it must be possible to use a
>> binary-compatible version of that module from a later release of
>> that implementation.
> 
> This is the only use of the term 'Java SE module'.  What is a Java SE 
> module?

Good catch; the document should define this term.  I'll fix that.

A "Java SE Module" is one that implements some portion of the Java SE
Platform Specification.  Such modules will themselves be part of that
Specification.  A typical platform implementation will contain other
modules as well, specific to the implementation.

> Am I correct in assuming this is describing _runtime_ upgradeable modules, 
> as clearly modules can be selected at build, link time in any case?

No, this describes modules in any phase, so that the overall model is
the same across phases.

> Why the distinction from other modules that may be upgradeable?  For 
> example, if I choose to issue a patched version of my java.base that is 
> binary compatible with a particular implementation of the platform, then I 
> would like to be able to upgrade that at runtime (see Referential 
> Integrity below).

Upgradeable modules are not meant for patching, and certainly not for
patching the base module.  They're meant for the small number of modules
which implement endorsed standards or standalone technologies and which
can, at least theoretically, evolve independently of Java SE itself
(though in practice, they rarely do).

More below ...

>> Referential integrity — Given a specific set of modules comprising
>> an implementation of the Java SE Platform, it must be possible to
>> ensure that each module can only be configured to reference other
>> modules in the same set, with the exception of upgradeable modules.
> 
> (I'm choosing to interpret 'can only be configured to reference' as 'can 
> be configured only to reference')

Right; I'll fix that.

> If I am interpreting this correctly it means that it must be possible, but 
> not strictly required, that there is no ad hoc runtime replacement of 
> modules in an SE image.  However, the set of permitted modules can be 
> arbitrarily defined to make runtime replacement possible.

Yes, insofar as that's both feasible and legitimate.

The modularization of the SE Platform will specify a set of modules, and
it will identify which must be upgradeable.  I suppose an implementor
could choose to make additional modules upgradeable, but that's an
implementation choice, it might run afoul of conformance issues, and in
some cases (e.g., the base module) it probably won't make much sense.

> For example, the VM can identify through naming, code signing, directory 
> location, or whatever, 'modules comprising an implementation of the Java 
> SE Platform' and refuse to load their dependencies if they are not in the 
> definition of that Platform set.  However, the 'defined set' is all 
> modules with a given name prefix, or signature, or directory location, etc 
> so I /can/ upgrade any modules comprising an implementation of the Java SE 
> Platform by providing a new binary compatible module that falls within 
> that set.
> 
> Is that interpretation valid.

Yes, I think you've more or less got it.

>                                How do you see this working in practice?

If you mean from the user's perspective, I expect the various phases
will support an `-upgrademodulepath` option, or its moral equivalent in
terms of APIs or configuration files, so that it's easy to specify newer
versions of upgradeable modules.

> Development
> 
>> Multi-mode artifacts — The existing JAR-file format must be enhanced
>> so that a module that does not contain native code can be delivered
>> in a single artifact that can be used either on the class path or as
>> a module.
> 
> Why the restriction on JAR files that contain native code?  Of course 
> plain old JAR files would not be able to take advantage of the native code 
> unless the application enabled this.  For example, OSGi allows for native 
> code to be delivered via application JAR files.  Consider changing to 
> "...so that a module that does not depend upon native code can be 
> delivered..."

I'm not sure what you're getting at here.  The JAR-file specification
does not describe how to include native code in a JAR file, and the
class-path mechanism does not support it.  The phrase "does not contain
native code" is meant to respect that precedent.  (It also, happily,
precludes the possibility that we'll extend the class-path mechanism to
support native code.)  Changing this to read "does not depend upon
native code" would, moreover, just make it more confusing, since all
Java code everywhere ultimately depends upon some native code.

OSGi bundles are, effectively, enhanced JAR files, and they can contain
native code, but if they do then they certainly won't work on the
regular old class path, so I don't see how they're relevant here.

> Dynamic Configuration
> 
> I am reading this section while making a clear distinction in my mind 
> between the Referential Integrity of the Java SE Platform modules, and the 
> more open nature of the modular application.
> 
> If this distinction is correct, then I feel we do need a clear 
> 'Fundamentals' definition of a Platform Module.  The module system manages 
> the Java Platform modules differently to the modules of a modular Java 
> application.

I think it would be unfortunate if the module system were to treat
Platform modules any differently than normal modules -- except for the
base module, which is special by definition.

Reading your comment makes me realize that the "referential integrity"
requirement is poorly worded.  It is motivated by SE Platform
modularization, but it's actually a more-general feature which could
well prove useful to other tightly-related collections of modules.
I'll reword it along those lines.

> Although described as Basic dynamic configuration, this section does 
> appear to open the door for runtime module dependency resolution.  This 
> dynamic configuration is carefully avoided for the Java SE Platform.

Run-time resolution must be possible, even for Platform modules, per the
"Run-time augmentation of platform modules" item.

- Mark