The future of partial builds

[Phil Race]

A huge step backwards.
I don't want to have to clone or keep hotspot up to date
and I prefer using the 'RE' builds of hotspot to any I would create.
I have never found this fragile. Its worked well for over 10 years ...

-phil.

On 9/10/2012 8:36 AM, Jonathan Gibbons wrote:
> Having to compile hotspot every time one creates a new repo seems like 
> a very significant step backwards.  I can clone and build langtools in 
> 45 seconds.
>
>> $ time ( hg clone http://hg.openjdk.java.net/jdk8/tl/langtools ; cd 
>> langtools ; lt.ant build-all-classes )
>> destination directory: langtools
>     ...
>> BUILD SUCCESSFUL
>> Total time: 23 seconds
>>
>> real    0m45.313s
>> user    0m23.909s
>> sys    0m12.461s
>
> -- Jon
>
>
>
> On 09/10/2012 07:20 AM, Magnus Ihse Bursie wrote:
>> On 2012-09-10 14:13, Alan Bateman wrote:
>>>
>>> I think this is a great topic to discuss.
>>>
>>> At least within Oracle then I think the majority of people do 
>>> partial builds in their local environment. When I say "partial 
>>> build" then I mean they build a subset of repositories, not all of 
>>> them. So folks working in the jdk repository tend to just build that 
>>> repository with everything else coming from whatever import JDK they 
>>> are using. Partial builds are of course very fragile and frequently 
>>> people have problems because they aren't using a matching import 
>>> JDK. Also periodically we have flag days, say changes that impact 
>>> langtools+jdk or hotspot+jdk and these usually cause a bit of 
>>> disruption until the changes get into a promoted build and into the 
>>> import JDK that people use. As to why people do partial builds then 
>>> it's probably a mix of issues including disk space, time to do the 
>>> first build, and of course it's just the way that we've always done it.
>>
>>
>> In build-infra, there is currently a "somewhat partial  build" 
>> feature that is implemented like this:
>> 1) You check out a "master forest", containing all repos. You only 
>> need to do this checkout once, and you are not required to 
>> pull/update it (unless you want to keep up with changes -- like on a 
>> flag day).
>>
>> 2) For subsequent work, you can clone just a single repo (let's call 
>> it "my_hotspot"), and doing some configure magic (currently a bit 
>> complicated, but that can be rewritten to be simpler) which will 
>> point to the "master forest" but replace the hotspot repo in the 
>> master forest with "my_hotspot"..
>>
>> 3) The first time you build "my_hotspot", it will rebuild the whole 
>> forest (but with hotspot replaced with my_hotspot). However, if you 
>> never pull/update the master forest, this will be a one-time event. 
>> Subsequent builds will see that nothing has changed in the master 
>> forest, and only changes in my_hotspot will be recompiled.
>>
>> The pros with this solution is:
>> 1) help with flag days -- just pull/update the master forest and 
>> rebuild.
>> 2) you can always build the complete product.
>> 3) you save some disk space by only cloning the complete forest once.
>>
>> However:
>> 1) you will still need disk space for build output for the complete 
>> forest, for every single repo.
>> 2) you will still need to recompile the whole forest for each new 
>> single repo you clone.
>>
>> Do you think this will make the solution not good enough?
>>
>> I think it is possible and not too hard to change this solution 
>> slightly, so you would only need to compile the master forest once, 
>> and then new single repos can use that build output as well as using 
>> the source code from the master forest. Though proper dependency 
>> checking might be harder, and there will be semantic issues about the 
>> meaning of things like "make clean" that can be surprising. Is this 
>> something you think is worth pursuing?
>>
>> To completely eliminate compiling the whole product at least once 
>> seems much harder. I can't really see a good way of using "import 
>> JDKs" the way they've been used in the old build system. I imagine 
>> one possibility is to prepare a tar ball, including a complete cloned 
>> forest (for a certain point in time) and a build directory with build 
>> artifacts corresponding to that source code and your platform(s) of 
>> choice, and have people download that as starting point. It seems 
>> like a lot of work, though. Do you think that is needed to get the 
>> new build system accepted?
>>
>> /Magnus
>
>
>