The future of partial builds

[Weijun Wang]

It's not uncommon that I just clone the repo(s), do a quick build, run 
several tests and abandon it, for example, if I want to try my code 
changes on a different platform (from my daily work machine). Therefore 
I find the jdk-only build very useful.

-Max

On 09/10/2012 10:20 PM, Magnus Ihse Bursie wrote:
> On 2012-09-10 14:13, Alan Bateman wrote:
>>
>> I think this is a great topic to discuss.
>>
>> At least within Oracle then I think the majority of people do partial
>> builds in their local environment. When I say "partial build" then I
>> mean they build a subset of repositories, not all of them. So folks
>> working in the jdk repository tend to just build that repository with
>> everything else coming from whatever import JDK they are using.
>> Partial builds are of course very fragile and frequently people have
>> problems because they aren't using a matching import JDK. Also
>> periodically we have flag days, say changes that impact langtools+jdk
>> or hotspot+jdk and these usually cause a bit of disruption until the
>> changes get into a promoted build and into the import JDK that people
>> use.  As to why people do partial builds then it's probably a mix of
>> issues including disk space, time to do the first build, and of course
>> it's just the way that we've always done it.
>
>
> In build-infra, there is currently a "somewhat partial  build" feature
> that is implemented like this:
> 1) You check out a "master forest", containing all repos. You only need
> to do this checkout once, and you are not required to pull/update it
> (unless you want to keep up with changes -- like on a flag day).
>
> 2) For subsequent work, you can clone just a single repo (let's call it
> "my_hotspot"), and doing some configure magic (currently a bit
> complicated, but that can be rewritten to be simpler) which will point
> to the "master forest" but replace the hotspot repo in the master forest
> with "my_hotspot"..
>
> 3) The first time you build "my_hotspot", it will rebuild the whole
> forest (but with hotspot replaced with my_hotspot). However, if you
> never pull/update the master forest, this will be a one-time event.
> Subsequent builds will see that nothing has changed in the master
> forest, and only changes in my_hotspot will be recompiled.
>
> The pros with this solution is:
> 1) help with flag days -- just pull/update the master forest and rebuild.
> 2) you can always build the complete product.
> 3) you save some disk space by only cloning the complete forest once.
>
> However:
> 1) you will still need disk space for build output for the complete
> forest, for every single repo.
> 2) you will still need to recompile the whole forest for each new single
> repo you clone.
>
> Do you think this will make the solution not good enough?
>
> I think it is possible and not too hard to change this solution
> slightly, so you would only need to compile the master forest once, and
> then new single repos can use that build output as well as using the
> source code from the master forest. Though proper dependency checking
> might be harder, and there will be semantic issues about the meaning of
> things like "make clean" that can be surprising. Is this something you
> think is worth pursuing?
>
> To completely eliminate compiling the whole product at least once seems
> much harder. I can't really see a good way of using "import JDKs" the
> way they've been used in the old build system. I imagine one possibility
> is to prepare a tar ball, including a complete cloned forest (for a
> certain point in time) and a build directory with build artifacts
> corresponding to that source code and your platform(s) of choice, and
> have people download that as starting point. It seems like a lot of
> work, though. Do you think that is needed to get the new build system
> accepted?
>
> /Magnus