Sharing experiences with the latest EA build // #ReflectiveAccessToNonExportedTypes #ResourceEncapsulation

[Rafael Winterhalter]

> With the newest EA build, out of 7000 unit tests, only
>> 42 tests still fail and those are all because of using non-exported API of
>> the JCL.
>>
> Just curious, what is "JCL" here?
>

I mean "Java class library", the classes that are part of the JVM.

This is of course not great as plain jars cannot be packaged
>> into jdeps but I hope to include a generated module-info.class at some
>> point even in a Java 6 jar in order to allow doing this.
>>
> I'm also curious what "plain jars cannot be packaged into jdeps" means. Do
> you mean the jdeps tool, maybe you mean jlink?
>

Yes, this is what I meant.


> Creating the
>> first module already helped to uncover a minor breach of the intended
>> module boundaries that was now discovered by the Java 9 compiler.
>>
> I assume these were warnings previously (as javac has emitted warnings
> since JDK 6 when it sees references to JDK-internal APIs).
>

I was using reflection for this functionality and offered a fallback or
offered a proper failure message. I did however manage to fix most of these
problems in the last hours by revisiting them. I am currently down to a
single failing unit test which relates to Class.forName(typeName, false,
classLoader) not returning its loaded classes if the class loader is an
instance of "DelegatingClassLoader". I worked around this by reading the
class containing vector for instances of this class loader and wrote a unit
tests that is no longer executable. As it only affects the test, this does
however not really matter.


> One
>> minor problem was that many times, several Maven modules declared the same
>> packages to communicate "cross-module shared classes" via package private
>> types. This required some refactoring of classes and splitting up of some
>> modules.
>>
> The split-package issue is going to challenge. In this case it sounds like
> it's all application and can be refactored without breaking others.
>
> Also there is also no shame is leaving some JARs on the class path.
>

Ideally, I would like to "jlink" some of my applications at a point. But
this is not a blocker.


>
>> After deploying the application with some first modules on the module
>> path,
>> I encountered similar problems. Adding correct exports to cover the use of
>> reflection is unfortunately non-trivial.
>>
> Can you expand a bit on this? Is this modules in this initial set
> reflecting on each other? If they are reflecting on other parts of the
> system then there should be no issue when they are weak or automatic
> modules, or just left on the class path.
>

I feel like using weak modules would only defer my migration problem to a
later point. What I would like to do would be to validate correct module
encapsulation only within the code I already managed to migrate to modules
without affecting the rest of my code base. This would allow me to migrate
code step by step. Currently, modularizing a single module changes the
semantics of reflecting on this module throughout my entire code base. With
about 80 libraries and 230 Maven modules, just validating a single module
is quite a bit of work and feels like a "big bang" for every module. What I
would like to do is to to change my code base incrementally only for the
code that is already aware of the module system.


> Step-by-step migration is
>> unfortunately impossible as I always need to consider all of our code and
>> the library code.
>>
> In the previous paragraph then it sounds like you were doing it
> incrementally but here it sounds like you ended moving all components, is
> that right? Did you trying using automatic modules or did you migrate
> everything to explicit modules ("explicit module" meaning it has a compiled
> module declaration).
>

I hope this is clear now from my above clarification. I was trying out
explicit, "proper" modules and added dependent jars as automatic modules.


> There is (obviously) no way to check at compile time if a
>> module is accessed refletively from the class path what only leaves
>> runtime. This is also true for non-reflective access if I do not recompile
>> all code on the class path when transforming a module. This turned out to
>> be quite tedious, especially, since some problems only occur when
>> triggering a particular subroutine of the application. Unit tests do not
>> offer a good alternative either as I cannot really separate what tests are
>> supposed to have access to the module internals and which should not. The
>> application has thousands of tests, some of them not making much sense to
>> me, going through them would be economically impractical. Integration
>> tests
>> sometimes uncover those access problems but unfortunately often disguise
>> them in the error reports.
>>
> If I read this correctly then everything was migrated to explicit and
> strong modules. I assume it would have been a lot easier to get going if
> using starting out with automatic (or even weak) modules.
>

Yes, this is what I did. I hope this is now clarified above.


> :
>>
>> I currently experiment with a Java agent to force Java 9 to adopt this
>> desired behavior by manipulating the boot layer's module graph or by
>> stubbing AccessibleObject::checkCanSetAccessible.
>>
> That sounds like a gross hack, something that could break at any time.
>

I am glad you like the idea. ;)


> Finally, I already identified some Maven modules which would be difficult
>> to Java-9-modularize as they basically only work with reflection and are
>> used throughout many parts of the application. It feels a bit strange to
>> export packages from the users of this module as those "reflection
>> modules"
>> are only used via several indirections. I would prefer it if I could
>> declare the module to be a form of "transitive module" which is simply
>> allowed to reflect as it was allowed previously. I understand that
>> security
>> might be an issue here but why not simply ask a potentially present
>> security manager if such a transitive module is allowed to be used?
>>
> With strong encapsulation then it's the module author that decides which
> parts of the module can be accessed by others. The weak modules proposal is
> a good starting point for application modules can take advantage of
> reliable configuration without opting in to strong encapsulation.
>

This would be a way to go but I am afraid that currently, the most likely
outcome would be to make all modules weak and keep them weak forever.
Personally, I believe that weak modules might become the default choice for
most developers as migration to proper module turns out so difficult and
would especially bind senior developers who tend to have the least amount
of time available.


>
> In my trial runs, I did not identify any problems with
>
>> #ResourceEncapsulation as all of our custom "cross-module files" were put
>> into the base folder, i.e. the unnamed module where they remain available.
>> I do however not know how this would affect other applications that put
>> resources into packages.
>>
> Thanks for confirming. It remains to be seen how much the ClassLoader APIs
> are used to get at non-class resources that are co-located with .class
> files.
>
> :
>>
>> PS: I noticed that when running java (without commands), it still lists
>> the
>> non-GNU-style options that are no longer supported.
>>
> Can you paste in `java -version` and give an example?
>

rafael at raftop:~$ ~/jdk-9/bin/java -version
java version "9-ea"
Java(TM) SE Runtime Environment (build
9-ea+138-jigsaw-nightly-h5546-20160929)
Java HotSpot(TM) 64-Bit Server VM (build
9-ea+138-jigsaw-nightly-h5546-20160929, mixed mode)

Running "java" does for example mention:

    -addmods <modulename>[,<modulename>...]

Which was replaced by the GNU-style "--add-modules".

Thanks for reading my experience report. I hope it will come in helpful!
Best regards, Rafael