The future of partial builds

[Magnus Ihse Bursie]

On 2012-09-10 14:13, Alan Bateman wrote:
>
> I think this is a great topic to discuss.
>
> At least within Oracle then I think the majority of people do partial 
> builds in their local environment. When I say "partial build" then I 
> mean they build a subset of repositories, not all of them. So folks 
> working in the jdk repository tend to just build that repository with 
> everything else coming from whatever import JDK they are using. 
> Partial builds are of course very fragile and frequently people have 
> problems because they aren't using a matching import JDK. Also 
> periodically we have flag days, say changes that impact langtools+jdk 
> or hotspot+jdk and these usually cause a bit of disruption until the 
> changes get into a promoted build and into the import JDK that people 
> use.  As to why people do partial builds then it's probably a mix of 
> issues including disk space, time to do the first build, and of course 
> it's just the way that we've always done it.


In build-infra, there is currently a "somewhat partial  build" feature 
that is implemented like this:
1) You check out a "master forest", containing all repos. You only need 
to do this checkout once, and you are not required to pull/update it 
(unless you want to keep up with changes -- like on a flag day).

2) For subsequent work, you can clone just a single repo (let's call it 
"my_hotspot"), and doing some configure magic (currently a bit 
complicated, but that can be rewritten to be simpler) which will point 
to the "master forest" but replace the hotspot repo in the master forest 
with "my_hotspot"..

3) The first time you build "my_hotspot", it will rebuild the whole 
forest (but with hotspot replaced with my_hotspot). However, if you 
never pull/update the master forest, this will be a one-time event. 
Subsequent builds will see that nothing has changed in the master 
forest, and only changes in my_hotspot will be recompiled.

The pros with this solution is:
1) help with flag days -- just pull/update the master forest and rebuild.
2) you can always build the complete product.
3) you save some disk space by only cloning the complete forest once.

However:
1) you will still need disk space for build output for the complete 
forest, for every single repo.
2) you will still need to recompile the whole forest for each new single 
repo you clone.

Do you think this will make the solution not good enough?

I think it is possible and not too hard to change this solution 
slightly, so you would only need to compile the master forest once, and 
then new single repos can use that build output as well as using the 
source code from the master forest. Though proper dependency checking 
might be harder, and there will be semantic issues about the meaning of 
things like "make clean" that can be surprising. Is this something you 
think is worth pursuing?

To completely eliminate compiling the whole product at least once seems 
much harder. I can't really see a good way of using "import JDKs" the 
way they've been used in the old build system. I imagine one possibility 
is to prepare a tar ball, including a complete cloned forest (for a 
certain point in time) and a build directory with build artifacts 
corresponding to that source code and your platform(s) of choice, and 
have people download that as starting point. It seems like a lot of 
work, though. Do you think that is needed to get the new build system 
accepted?

/Magnus