Switching to split repos

[Doug Simon]

To better reflect the different technologies within the Graal stack, we will soon be switching to disjoint repositories. This will allow one to focus on a smaller code base and treat other parts of the stack as external dependencies.

The new repositories replacing http://hg.openjdk.java.net/graal/graal (“old graal") are: 

http://hg.openjdk.java.net/graal/graal-jvmci-8  (JVMCI and HotSpot code from old graal based on JDK8)
http://hg.openjdk.java.net/graal/graal-jvmci-9  (JVMCI and HotSpot code from old graal based on JDK9)
http://hg.openjdk.java.net/graal/graal-compiler (old graal minus graal-jvmci-8)
https://bitbucket.org/allr/mx                   (updated, standalone mx that works with split repos)

The http://hg.openjdk.java.net/graal/graal has already been removed but its mirror at https://github.com/OracleLabs/GraalVM <https://github.com/OracleLabs/GraalVM is still live.

While there is already content mirrored at these repos, do not pull from them (unless you want untested changes that may not yet work). A follow-up email will be sent (within the next week) once you can pull from them.

The first step in switching to the split repos is to pull mx:

$ hg clone https://bitbucket.org/allr/mx

Since mx is now a standalone tool, it locates the ‘primary’ suite (i.e. the repo you are primarily working on) based on the current working directory instead of the location of mx. If you are within a repo directory hierarchy, that repo is the primary suite. Otherwise, you will need to use the —primary-suite-path or -p option.

By default, mx will pull dependent repos and place them as siblings of your primary repo so it’s highly recommended to start in an empty directory. For example:

$ mkdir graal-compiler-workspace
$ cd graal-compiler-workspace
$ mx sclone http://hg.openjdk.java.net/graal/graal-compiler graal
$ ls -1
graal
jvmci

Note the directory names are determined by the information in suite.py for each suite, not by the URL it was pulled from (except for the primary suite where the name or URL given to the sclone command is used).

Mx has some new commands that simplify working with split repos. These are briefly described below in terms of changes to standard workflows. More extensive documentation is at https://bitbucket.org/allr/mxtool2/wiki/Home.

1. Update to the newest version

old:
$ hg pull
new:
$ mx spull

2. Make a cross-repo change

in the jvmci repo:
do the changes
$ hg commit -m "..."
in the graal repo:
$ mx scheckimports
do the merge fixes (if any)
$ hg commit -m "update imports"

Mx also imposes some changes on suite.py files

1. There is an “imports” section (which should be mostly self explanatory) for specifying the suite dependencies.
2. Inter-suite references can now only be to distributions and libraries and must be prefixed with the suite name (e.g., “jvmci:JVMCI_API”).
3. Annotation processors (APs) must now be distributions (see 2.).
4. A distribution must explicitly declare all of the distributions it depends on in its “distDependencies” attribute.

These constraints make suite.py files more intuitive. Some of them are also necessary to support importing suites as binaries. Binary suites allow one to use pre-built binary versions of the dependencies of a repository. This can be useful when only read access to the sources is required.

Currently, importing source suites is the default. Using binary dependencies instead can be enabled using the MX_BINARY_SUITES environment variable (which can be persisted in mx.<primary suite name>/env). This is a comma-separated list of the suites for which binaries should be used. For example:

MX_BINARY_SUITES=jvmci

One other change is that mx no longer looks for ecj.jar in the mx dir of the primary suite. It needs to be explicitly supplied with the --jdt option to the build command or with the JDT environment variable.

While we will try to ensure everything is working on the switch over date, there’s bound to be a few bumps in during this transition. The best way to get issues resolved is to ask on this list (graal-dev at openjdk.java.net).

-Doug