Getting a live view of environment variables (Gradle and JDK 9)

[Cédric Champeau]

Hi all,

I'm writing this on behalf of the Gradle team. This email is closely
related to the other thread just posted today, but just a timeline
coincidence (just like the email exchange I had today about this with Alan
Bateman ;)) and not exactly the same issue.

We are in the process of making sure Gradle runs properly on JDK 9, but
there's an issue which is unresolved so far, and probably requires a new
API. It's described at [1], and I have discussed this at Devoxx France with
Rémi Forax who suggested to post something here.

In short, Gradle is a build tool which supports building a variety of
different things, from Java to C++. The JVM happens to be its runtime
environment, and Gradle has what we call the Gradle Daemon [2] which is a
long running process, benefiting from the JIT, aimed at effectively running
builds. When the build starts, a client connects to the daemon and sends a
"build request". A daemon will run a single build at a time, and there are
several cases which would trigger a new daemon to spawn (the daemon JVM
arguments are one) but the environment variables are *not*. Since the
daemon is a long running process, it is possible that the environment
variables are mutated between the moment the daemon was spawned (in a
previous build) and the moment the build is executed.

What we do, now, is to send the environment variables of the client to the
daemon, which *mutates* the existing environment variables map provided by
System.getenv. This is exactly what is described in [1] as being sneaky (it
is) and broken in JDK 9 (since the underlying map doesn't exist anymore).
However, there are valid use cases for this:

   - in practice, environment variables are not immutable. It is especially
   true for long running process.
   - native programs can mutate the environment variables. Even if it's not
   recommended, it is possible and legal.
   - Gradle runs in a "blackbox": we don't know what plugins are doing.
   Even if we provide an API which gives access to "environment variables",
   and that those environment variables are not the ones returned by
   System.getenv, plugin authors would have to use this new API to get
   correct information. However, they may use libraries which access
   System.getenv directly, or use native APIs which would get out-of-sync
   information.
   - we need to propagate the environment to forked process (typically,
   forked compilers and worker daemons)

This means that today, we use JNI to effectively mutate the environment
variables of running process (that’s one of the purposes of the
native-platform project). Then, we mutate the backing map of the JDK to
reflect those changes, otherwise the mutation is not visible from Java code.

What can we do now?

   - Have the JDK honor the fact that environment variables *can* be
   mutated, because it just happens. In short, don't create an immutable copy
   of environment variables at startup, but provide a live view of the
   environment variables (using the existing APIs, System.getenv, would be the
   best thing because it would be immediately visible to all consumers,
   including 3rd party code run in plugins). In addition (but not mandatory),
   you could provide us with an API to set environment variables directly from
   Java. This would avoid JNI calls to do this. However, it’s not mandatory,
   because the live view of environment variables would just work in this case.
   - Last, but we would really, really avoid to do this, spawn a new daemon
   if we detect that the environment variables have changed (diff between what
   the client has and the daemon sees). The major drawback of this approach is
   that it kills performance, since a new daemon would have to be spawned. And
   it is likely to do so each time something (through native code, for
   example), mutates environment variables. A very simple example is
the PWD environment
   variables on Linux which contains the working directory. Basically changing
   the directory would be enough to spawn a new daemon. Another example is the
   TERM_SESSION_ID one, which means that 2 different terminals would force
   us to spawn 2 different Gradle daemons. We could, of course, have a list of
   “blessed” environments variables that we don’t trust, but it’s very easily
   broken, and no good design. That’s why, even if it’s possible, we don’t
   consider this a solution.

Thanks for considering our request, which is currently a blocker for us
(understand, have Gradle running properly under JDK 9).

[1] https://github.com/adammurdoch/native-platform/issues/16
[2] https://docs.gradle.org/current/userguide/gradle_daemon.html