The future of partial builds

[Jonathan Gibbons]

Having to compile hotspot every time one creates a new repo seems like a 
very significant step backwards.  I can clone and build langtools in 45 
seconds.

> $ time ( hg clone http://hg.openjdk.java.net/jdk8/tl/langtools ; cd 
> langtools ; lt.ant build-all-classes )
> destination directory: langtools
     ...
> BUILD SUCCESSFUL
> Total time: 23 seconds
>
> real    0m45.313s
> user    0m23.909s
> sys    0m12.461s

-- Jon



On 09/10/2012 07:20 AM, Magnus Ihse Bursie wrote:
> On 2012-09-10 14:13, Alan Bateman wrote:
>>
>> I think this is a great topic to discuss.
>>
>> At least within Oracle then I think the majority of people do partial 
>> builds in their local environment. When I say "partial build" then I 
>> mean they build a subset of repositories, not all of them. So folks 
>> working in the jdk repository tend to just build that repository with 
>> everything else coming from whatever import JDK they are using. 
>> Partial builds are of course very fragile and frequently people have 
>> problems because they aren't using a matching import JDK. Also 
>> periodically we have flag days, say changes that impact langtools+jdk 
>> or hotspot+jdk and these usually cause a bit of disruption until the 
>> changes get into a promoted build and into the import JDK that people 
>> use. As to why people do partial builds then it's probably a mix of 
>> issues including disk space, time to do the first build, and of 
>> course it's just the way that we've always done it.
>
>
> In build-infra, there is currently a "somewhat partial  build" feature 
> that is implemented like this:
> 1) You check out a "master forest", containing all repos. You only 
> need to do this checkout once, and you are not required to pull/update 
> it (unless you want to keep up with changes -- like on a flag day).
>
> 2) For subsequent work, you can clone just a single repo (let's call 
> it "my_hotspot"), and doing some configure magic (currently a bit 
> complicated, but that can be rewritten to be simpler) which will point 
> to the "master forest" but replace the hotspot repo in the master 
> forest with "my_hotspot"..
>
> 3) The first time you build "my_hotspot", it will rebuild the whole 
> forest (but with hotspot replaced with my_hotspot). However, if you 
> never pull/update the master forest, this will be a one-time event. 
> Subsequent builds will see that nothing has changed in the master 
> forest, and only changes in my_hotspot will be recompiled.
>
> The pros with this solution is:
> 1) help with flag days -- just pull/update the master forest and rebuild.
> 2) you can always build the complete product.
> 3) you save some disk space by only cloning the complete forest once.
>
> However:
> 1) you will still need disk space for build output for the complete 
> forest, for every single repo.
> 2) you will still need to recompile the whole forest for each new 
> single repo you clone.
>
> Do you think this will make the solution not good enough?
>
> I think it is possible and not too hard to change this solution 
> slightly, so you would only need to compile the master forest once, 
> and then new single repos can use that build output as well as using 
> the source code from the master forest. Though proper dependency 
> checking might be harder, and there will be semantic issues about the 
> meaning of things like "make clean" that can be surprising. Is this 
> something you think is worth pursuing?
>
> To completely eliminate compiling the whole product at least once 
> seems much harder. I can't really see a good way of using "import 
> JDKs" the way they've been used in the old build system. I imagine one 
> possibility is to prepare a tar ball, including a complete cloned 
> forest (for a certain point in time) and a build directory with build 
> artifacts corresponding to that source code and your platform(s) of 
> choice, and have people download that as starting point. It seems like 
> a lot of work, though. Do you think that is needed to get the new 
> build system accepted?
>
> /Magnus