Getting a live view of environment variables (Gradle and JDK 9)

[Martin Buchholz]

It seems to be the day for environment variable mutation.

System.getenv was always designed as an immutable snapshot, and more
recently there's emerging consensus that one must never ever mutate
environment variables in a multi-threaded Unix program.  One tends to
acquire a biased view on this after chasing down production programs
mysteriously crashing in getenv.

https://bugs.openjdk.java.net/browse/JDK-8173654
https://bugs.openjdk.java.net/browse/JDK-4953367

On Wed, May 10, 2017 at 2:37 PM, Cédric Champeau <cedric.champeau at gmail.com>
wrote:

> Hi all,
>
> I'm writing this on behalf of the Gradle team. This email is closely
> related to the other thread just posted today, but just a timeline
> coincidence (just like the email exchange I had today about this with Alan
> Bateman ;)) and not exactly the same issue.
>
> We are in the process of making sure Gradle runs properly on JDK 9, but
> there's an issue which is unresolved so far, and probably requires a new
> API. It's described at [1], and I have discussed this at Devoxx France with
> Rémi Forax who suggested to post something here.
>
> In short, Gradle is a build tool which supports building a variety of
> different things, from Java to C++. The JVM happens to be its runtime
> environment, and Gradle has what we call the Gradle Daemon [2] which is a
> long running process, benefiting from the JIT, aimed at effectively running
> builds. When the build starts, a client connects to the daemon and sends a
> "build request". A daemon will run a single build at a time, and there are
> several cases which would trigger a new daemon to spawn (the daemon JVM
> arguments are one) but the environment variables are *not*. Since the
> daemon is a long running process, it is possible that the environment
> variables are mutated between the moment the daemon was spawned (in a
> previous build) and the moment the build is executed.
>
> What we do, now, is to send the environment variables of the client to the
> daemon, which *mutates* the existing environment variables map provided by
> System.getenv. This is exactly what is described in [1] as being sneaky (it
> is) and broken in JDK 9 (since the underlying map doesn't exist anymore).
> However, there are valid use cases for this:
>
>    - in practice, environment variables are not immutable. It is especially
>    true for long running process.
>    - native programs can mutate the environment variables. Even if it's not
>    recommended, it is possible and legal.
>    - Gradle runs in a "blackbox": we don't know what plugins are doing.
>    Even if we provide an API which gives access to "environment variables",
>    and that those environment variables are not the ones returned by
>    System.getenv, plugin authors would have to use this new API to get
>    correct information. However, they may use libraries which access
>    System.getenv directly, or use native APIs which would get out-of-sync
>    information.
>    - we need to propagate the environment to forked process (typically,
>    forked compilers and worker daemons)
>
> This means that today, we use JNI to effectively mutate the environment
> variables of running process (that’s one of the purposes of the
> native-platform project). Then, we mutate the backing map of the JDK to
> reflect those changes, otherwise the mutation is not visible from Java
> code.
>
> What can we do now?
>
>    - Have the JDK honor the fact that environment variables *can* be
>    mutated, because it just happens. In short, don't create an immutable
> copy
>    of environment variables at startup, but provide a live view of the
>    environment variables (using the existing APIs, System.getenv, would be
> the
>    best thing because it would be immediately visible to all consumers,
>    including 3rd party code run in plugins). In addition (but not
> mandatory),
>    you could provide us with an API to set environment variables directly
> from
>    Java. This would avoid JNI calls to do this. However, it’s not
> mandatory,
>    because the live view of environment variables would just work in this
> case.
>    - Last, but we would really, really avoid to do this, spawn a new daemon
>    if we detect that the environment variables have changed (diff between
> what
>    the client has and the daemon sees). The major drawback of this
> approach is
>    that it kills performance, since a new daemon would have to be spawned.
> And
>    it is likely to do so each time something (through native code, for
>    example), mutates environment variables. A very simple example is
> the PWD environment
>    variables on Linux which contains the working directory. Basically
> changing
>    the directory would be enough to spawn a new daemon. Another example is
> the
>    TERM_SESSION_ID one, which means that 2 different terminals would force
>    us to spawn 2 different Gradle daemons. We could, of course, have a
> list of
>    “blessed” environments variables that we don’t trust, but it’s very
> easily
>    broken, and no good design. That’s why, even if it’s possible, we don’t
>    consider this a solution.
>
> Thanks for considering our request, which is currently a blocker for us
> (understand, have Gradle running properly under JDK 9).
>
> [1] https://github.com/adammurdoch/native-platform/issues/16
> [2] https://docs.gradle.org/current/userguide/gradle_daemon.html
>