Getting a live view of environment variables (Gradle and JDK 9)

[David Holmes]

Hi Cedric,

Like Martin I am somewhat surprised by the recent surge in interest in 
the JVM environment :)

 From a pragmatic perspective it is far too late to do anything about 
this in 9.

I am curious what kind of environment variables are being set by the 
client, how they are shared with the daemon, and what code actually 
reads their values?

As Martin suggests your only recourse may be to do everything via JNI.

David

On 11/05/2017 7:37 AM, Cédric Champeau wrote:
> Hi all,
>
> I'm writing this on behalf of the Gradle team. This email is closely
> related to the other thread just posted today, but just a timeline
> coincidence (just like the email exchange I had today about this with Alan
> Bateman ;)) and not exactly the same issue.
>
> We are in the process of making sure Gradle runs properly on JDK 9, but
> there's an issue which is unresolved so far, and probably requires a new
> API. It's described at [1], and I have discussed this at Devoxx France with
> Rémi Forax who suggested to post something here.
>
> In short, Gradle is a build tool which supports building a variety of
> different things, from Java to C++. The JVM happens to be its runtime
> environment, and Gradle has what we call the Gradle Daemon [2] which is a
> long running process, benefiting from the JIT, aimed at effectively running
> builds. When the build starts, a client connects to the daemon and sends a
> "build request". A daemon will run a single build at a time, and there are
> several cases which would trigger a new daemon to spawn (the daemon JVM
> arguments are one) but the environment variables are *not*. Since the
> daemon is a long running process, it is possible that the environment
> variables are mutated between the moment the daemon was spawned (in a
> previous build) and the moment the build is executed.
>
> What we do, now, is to send the environment variables of the client to the
> daemon, which *mutates* the existing environment variables map provided by
> System.getenv. This is exactly what is described in [1] as being sneaky (it
> is) and broken in JDK 9 (since the underlying map doesn't exist anymore).
> However, there are valid use cases for this:
>
>    - in practice, environment variables are not immutable. It is especially
>    true for long running process.
>    - native programs can mutate the environment variables. Even if it's not
>    recommended, it is possible and legal.
>    - Gradle runs in a "blackbox": we don't know what plugins are doing.
>    Even if we provide an API which gives access to "environment variables",
>    and that those environment variables are not the ones returned by
>    System.getenv, plugin authors would have to use this new API to get
>    correct information. However, they may use libraries which access
>    System.getenv directly, or use native APIs which would get out-of-sync
>    information.
>    - we need to propagate the environment to forked process (typically,
>    forked compilers and worker daemons)
>
> This means that today, we use JNI to effectively mutate the environment
> variables of running process (that’s one of the purposes of the
> native-platform project). Then, we mutate the backing map of the JDK to
> reflect those changes, otherwise the mutation is not visible from Java code.
>
> What can we do now?
>
>    - Have the JDK honor the fact that environment variables *can* be
>    mutated, because it just happens. In short, don't create an immutable copy
>    of environment variables at startup, but provide a live view of the
>    environment variables (using the existing APIs, System.getenv, would be the
>    best thing because it would be immediately visible to all consumers,
>    including 3rd party code run in plugins). In addition (but not mandatory),
>    you could provide us with an API to set environment variables directly from
>    Java. This would avoid JNI calls to do this. However, it’s not mandatory,
>    because the live view of environment variables would just work in this case.
>    - Last, but we would really, really avoid to do this, spawn a new daemon
>    if we detect that the environment variables have changed (diff between what
>    the client has and the daemon sees). The major drawback of this approach is
>    that it kills performance, since a new daemon would have to be spawned. And
>    it is likely to do so each time something (through native code, for
>    example), mutates environment variables. A very simple example is
> the PWD environment
>    variables on Linux which contains the working directory. Basically changing
>    the directory would be enough to spawn a new daemon. Another example is the
>    TERM_SESSION_ID one, which means that 2 different terminals would force
>    us to spawn 2 different Gradle daemons. We could, of course, have a list of
>    “blessed” environments variables that we don’t trust, but it’s very easily
>    broken, and no good design. That’s why, even if it’s possible, we don’t
>    consider this a solution.
>
> Thanks for considering our request, which is currently a blocker for us
> (understand, have Gradle running properly under JDK 9).
>
> [1] https://github.com/adammurdoch/native-platform/issues/16
> [2] https://docs.gradle.org/current/userguide/gradle_daemon.html
>