Switching to split repos

[Doug Simon]

Another follow up:

The ‘mx build’ command now does deeper checking between dependencies when determining what needs to be (re)built. For a given project or distribution A, it looks at the last modification timestamp for each of A’s transitive dependencies (in addition to A’s own sources). If a timestamp is found that is more recent than the timestamp of any of A’s sources, A is rebuilt. This granularity of checking does not work so well with the fine grained incremental building done by Eclipse. For example:

Project A
  X.java
  Y.java
Project B (depends on A)
  Z.java (depends on X)

1. Edit Y.java in Eclipse, save
2. Eclipse rebuilds Y and is smart enough to understand that Z does not need to be rebuilt since it has no dependencies on Y
3. Run mx.build
  - it looks at project A, all class files are up to date, nothing to do
  - it looks at project B, Z.class is up to date wrt Z.java, but it is older than Y.class so B is rebuilt by mx.

While this conservative behavior always produces the correct results, it can potentially do a lot of redundant rebuilding. If you work in Eclipse and want the old behavior of ‘mx build’ where this extra dependency checking was not done, there is a new[1] -s (or --shallow-dependency-checks) flag you can use:

dsimon at kurz ~/g/graal-jvmci-8> mx build -h
...
  -s, --shallow-dependency-checks
                        ignore modification times of output files for each of
                        P's dependencies when determining if P should be
                        built. That is, only P's sources, suite.py of its
                        suite and whether any of P's dependencies have been
                        built are considered. This is useful when an external
                        tool (such as an Eclipse) performs incremental
                        compilation that produces finer grained modification
                        times than mx's build system. Shallow dependency
                        checking only applies to non-native projects.

-Doug

[1] https://bitbucket.org/allr/mx/commits/e64195713dff4888e07ee5a3fd465eae683904e9

> On 12 Aug 2015, at 11:04, Benoit Daloze <eregontp at gmail.com> wrote:
> 
> A little follow-up on the instructions, once the `mx sclone` is done:
> 
> $ cd graal
> $ mx build
> 
> The resulting build is in ../jvmci/jdk*/product
> 
> On Fri, Jul 24, 2015 at 4:27 PM, Doug Simon <doug.simon at oracle.com> wrote:
> To better reflect the different technologies within the Graal stack, we will soon be switching to disjoint repositories. This will allow one to focus on a smaller code base and treat other parts of the stack as external dependencies.
> 
> The new repositories replacing http://hg.openjdk.java.net/graal/graal (“old graal") are:
> 
> http://hg.openjdk.java.net/graal/graal-jvmci-8  (JVMCI and HotSpot code from old graal based on JDK8)
> http://hg.openjdk.java.net/graal/graal-jvmci-9  (JVMCI and HotSpot code from old graal based on JDK9)
> http://hg.openjdk.java.net/graal/graal-compiler (old graal minus graal-jvmci-8)
> https://bitbucket.org/allr/mx                   (updated, standalone mx that works with split repos)
> 
> The http://hg.openjdk.java.net/graal/graal has already been removed but its mirror at https://github.com/OracleLabs/GraalVM <https://github.com/OracleLabs/GraalVM is still live.
> 
> While there is already content mirrored at these repos, do not pull from them (unless you want untested changes that may not yet work). A follow-up email will be sent (within the next week) once you can pull from them.
> 
> The first step in switching to the split repos is to pull mx:
> 
> $ hg clone https://bitbucket.org/allr/mx
> 
> Since mx is now a standalone tool, it locates the ‘primary’ suite (i.e. the repo you are primarily working on) based on the current working directory instead of the location of mx. If you are within a repo directory hierarchy, that repo is the primary suite. Otherwise, you will need to use the —primary-suite-path or -p option.
> 
> By default, mx will pull dependent repos and place them as siblings of your primary repo so it’s highly recommended to start in an empty directory. For example:
> 
> $ mkdir graal-compiler-workspace
> $ cd graal-compiler-workspace
> $ mx sclone http://hg.openjdk.java.net/graal/graal-compiler graal
> $ ls -1
> graal
> jvmci
> 
> Note the directory names are determined by the information in suite.py for each suite, not by the URL it was pulled from (except for the primary suite where the name or URL given to the sclone command is used).
> 
> Mx has some new commands that simplify working with split repos. These are briefly described below in terms of changes to standard workflows. More extensive documentation is at https://bitbucket.org/allr/mxtool2/wiki/Home.
> 
> 1. Update to the newest version
> 
> old:
> $ hg pull
> new:
> $ mx spull
> 
> 2. Make a cross-repo change
> 
> in the jvmci repo:
> do the changes
> $ hg commit -m "..."
> in the graal repo:
> $ mx scheckimports
> do the merge fixes (if any)
> $ hg commit -m "update imports"
> 
> Mx also imposes some changes on suite.py files
> 
> 1. There is an “imports” section (which should be mostly self explanatory) for specifying the suite dependencies.
> 2. Inter-suite references can now only be to distributions and libraries and must be prefixed with the suite name (e.g., “jvmci:JVMCI_API”).
> 3. Annotation processors (APs) must now be distributions (see 2.).
> 4. A distribution must explicitly declare all of the distributions it depends on in its “distDependencies” attribute.
> 
> These constraints make suite.py files more intuitive. Some of them are also necessary to support importing suites as binaries. Binary suites allow one to use pre-built binary versions of the dependencies of a repository. This can be useful when only read access to the sources is required.
> 
> Currently, importing source suites is the default. Using binary dependencies instead can be enabled using the MX_BINARY_SUITES environment variable (which can be persisted in mx.<primary suite name>/env). This is a comma-separated list of the suites for which binaries should be used. For example:
> 
> MX_BINARY_SUITES=jvmci
> 
> One other change is that mx no longer looks for ecj.jar in the mx dir of the primary suite. It needs to be explicitly supplied with the --jdt option to the build command or with the JDT environment variable.
> 
> While we will try to ensure everything is working on the switch over date, there’s bound to be a few bumps in during this transition. The best way to get issues resolved is to ask on this list (graal-dev at openjdk.java.net).
> 
> -Doug
>