Sharing experiences with the latest EA build // #ReflectiveAccessToNonExportedTypes #ResourceEncapsulation

Wed Oct 5 11:22:31 UTC 2016

 I was asked in a private email how I got an enterprise application to run
on the system class loader as WAR/EAR-packaging (currently) defeats the
module system at runtime. Therefore, I wanted to quickly add the following
advice if anybody wants to try out their own application:

I had to tweak the application a bit; it is typically bundled in a "war"
file but all we do this for is to deploy upon the servlet API. What I
therefore did was to skip this last bundling step of our build. Instead, I
wrote a small starter module which runs Jetty programmatically while
explicitly registering the application's servlets. This way, I could use
modules on the system class loader rather than including any dependencies
in the war file.

2016-09-30 14:35 GMT+02:00 Rafael Winterhalter <rafael.wth at gmail.com>:

> I have tested the latest EA builds, both on my open-source projects and in
> several production applications, one of them quite large with thousands of
> development hours invested. I wanted to share my experiences.
>
> After making files with the .class extension visible, my agent/code
> generation library Byte Buddy works more or less as it did before.
> Previously, I had to apply a work-around where I mapped package names of
> all modules to their Module instance what allowed me read class files via
> this instance. I discovered all modules via Layer.boot() but was of course
> missing support for modules of other layers. This approach also hurt
> runtime performance. With the newest EA build, out of 7000 unit tests, only
> 42 tests still fail and those are all because of using non-exported API of
> the JCL. None of those usages are essential to using the library such that
> I can live with this minor limitation. (I did not yet look to adding a file
> in the META-INF folder which I read somewhere was possible.) I can also
> confirm that using the latest EA build, cglib and Javassist work again.
> This is great news as this breaking change really concerned me with regards
> to Java 9 compatibility. Code generation libraries are used everywhere and
> breaking them would have had an impact on any user of the JVM. As I
> currently support Java 6, I do not plan to apply modules on my OSS in the
> near future. This is of course not great as plain jars cannot be packaged
> into jdeps but I hope to include a generated module-info.class at some
> point even in a Java 6 jar in order to allow doing this.
>
> I also tried to migrate a small and a large production application. The
> larger applications is currently running on a Java 8 VM and code level but
> started out in the days of Java 4. As the project is already organized in
> Maven modules, approaching the migration was straight-forward. Creating the
> first module already helped to uncover a minor breach of the intended
> module boundaries that was now discovered by the Java 9 compiler. The
> modularization did however not pay off too much due to the previous
> modularization where the Maven modularization already offered a good
> approximation. Alltogether, my experience was however quite positive. One
> minor problem was that many times, several Maven modules declared the same
> packages to communicate "cross-module shared classes" via package private
> types. This required some refactoring of classes and splitting up of some
> modules.
>
> Getting the application to run was however not as easy as the adoption of
> compile-time modules. As mentioned, the application has grown quite big
> over the years and has many dependencies without anybody having a complete
> overview of what is used and for what. In this sense, the application is
> not polished but runs quite robust in production. (In the end, I think the
> application is quite representative of many commercial applications out in
> the wild.) Using Java 9 but without applying modules, the application did
> start but sometimes fails at runtime due to libraries and code accessing
> non-exported classes of the JCL. Many of these breaches happen high up the
> stack in library and framework code. For now, I have therefore stripped
> down the application to avoid using those application parts which I do not
> know intimately enough to fix right now.
>
> After deploying the application with some first modules on the module
> path, I encountered similar problems. Adding correct exports to cover the
> use of reflection is unfortunately non-trivial. My IDE lists over 420
> usages of Method::invoke in the project and its libraries what makes it
> practically impossible to trace all possible reflective access. Generally,
> I do neither want to depend too much on such implementation details as it
> makes updating dependencies quite tedious as I have to revalidate the
> access patterns. My major problem with adding additional modules onto the
> module path is that each breaks code that is still on the class path.
> Step-by-step migration is unfortunately impossible as I always need to
> consider all of our code and the library code. There is (obviously) no way
> to check at compile time if a module is accessed refletively from the class
> path what only leaves runtime. This is also true for non-reflective access
> if I do not recompile all code on the class path when transforming a
> module. This turned out to be quite tedious, especially, since some
> problems only occur when triggering a particular subroutine of the
> application. Unit tests do not offer a good alternative either as I cannot
> really separate what tests are supposed to have access to the module
> internals and which should not. The application has thousands of tests,
> some of them not making much sense to me, going through them would be
> economically impractical. Integration tests sometimes uncover those access
> problems but unfortunately often disguise them in the error reports.
>
> To be fair, I think that I would get the application going if I invested
> more time to trial and error. However, I still find the migration process
> too difficult as it requires a lot of manual runs to validate. I think it
> would be helpful if code on the class path would behave exactly as before
> to allow for a better step-by-step migration, i.e. could access modularized
> code reflectively without any constraints. Right now, it feels easiest to
> export about everything before I add something to the module path.
> Otherwise, it does not feel safe to apply the migration when I think about
> putting the application into production: even when the application appears
> to be running, errors can always occur later when a specific unit of code
> is triggered. I am mostly scared when thinking of rarely triggered routines
> like security failovers or error reporting mechanisms which are often
> implemented quite generically (reads: use a lot of reflection).
>
> I currently experiment with a Java agent to force Java 9 to adopt this
> desired behavior by manipulating the boot layer's module graph or by
> stubbing AccessibleObject::checkCanSetAccessible. I hope to trial this a
> bit broader soon but it seems promising to allow us a "safe migration" to
> Java 9. I would of course prefer to remove this agent at some point in time
> but I would not know when or if all of our dependencies are "Java 9 ready".
> (Some of them have not been updated for years.) I am also doing this as a
> means of delegating migration to developers with less Java experience as
> migrating to Java 9 turns out to be quite a demanding task. To be honest,
> many junior developers do not even know what reflection is and it would be
> difficult to explain to them how they need to modularize the code they work
> with. As many libraries (or our own modules) do not document their
> implementation but only their API, the only way to figure out the module
> requirements is by reading its source code. For framework code, this often
> outchallanges less experienced developers.
>
> Finally, I already identified some Maven modules which would be difficult
> to Java-9-modularize as they basically only work with reflection and are
> used throughout many parts of the application. It feels a bit strange to
> export packages from the users of this module as those "reflection modules"
> are only used via several indirections. I would prefer it if I could
> declare the module to be a form of "transitive module" which is simply
> allowed to reflect as it was allowed previously. I understand that security
> might be an issue here but why not simply ask a potentially present
> security manager if such a transitive module is allowed to be used?
>
> My conclusion is that I wished that migration would be easier to do
> step-by-step. This way I can better spread the costs of adoption.
> Currently, I think it would be easiest to not migrate to modules as the
> benefits are limited in the context of existing Maven modularization. Right
> now, any introduced Java module requires a lot of additional runtime
> testing and I doubt that my customers would be willing to pay for it. With
> a greenfield project, I would always try to adopt modules as they push
> people into the right direction but as long as the library landscape is not
> modularized, this again bears a high risk of runtime errors if those
> libraries are not module aware. As many libraries are careful about
> updating to not alienate legacy VM users, I imagine that this might take a
> while. As for "hard updates", I still think a "force-exporting" Java agent
> is a solution that I consider, simply to run on a Java 9 VM without having
> to change or validate any code. There are a lot of great features that I
> hope to take advantage of, besides Jigsaw.
>
> Even though I know this idea is not popular, my recommendation is to have
> the class path to behave just as it did before, i.e. reflection can cross
> module boundaries without preparing those modules. This especially in the
> context of the impossibility of putting the JCL onto the class path.
>
> In my trial runs, I did not identify any problems with
> #ResourceEncapsulation as all of our custom "cross-module files" were put
> into the base folder, i.e. the unnamed module where they remain available.
> I do however not know how this would affect other applications that put
> resources into packages.
>
> So much for my experience report. Altogether, I really think Java 9 has a
> lot to offer and I really appreciate Oracle is not rushing this but takes
> the time it takes to getting this right. Great work, also beyond Jigsaw
> there is much I am looking forward too.
>
> Best regards, Rafael
>
> PS: I noticed that when running java (without commands), it still lists
> the non-GNU-style options that are no longer supported.
>