The future of partial builds

[Weijun Wang]

Sorry I was not clear, by "clone the repo(s)", I mean either only the 
jdk repo, or plus the jdk/*/closed ones. I almost never clone other 
repos (langtools, hotspot, ...).

-Max

On 09/10/2012 10:48 PM, Weijun Wang wrote:
> It's not uncommon that I just clone the repo(s), do a quick build, run
> several tests and abandon it, for example, if I want to try my code
> changes on a different platform (from my daily work machine). Therefore
> I find the jdk-only build very useful.
>
> -Max
>
> On 09/10/2012 10:20 PM, Magnus Ihse Bursie wrote:
>> On 2012-09-10 14:13, Alan Bateman wrote:
>>>
>>> I think this is a great topic to discuss.
>>>
>>> At least within Oracle then I think the majority of people do partial
>>> builds in their local environment. When I say "partial build" then I
>>> mean they build a subset of repositories, not all of them. So folks
>>> working in the jdk repository tend to just build that repository with
>>> everything else coming from whatever import JDK they are using.
>>> Partial builds are of course very fragile and frequently people have
>>> problems because they aren't using a matching import JDK. Also
>>> periodically we have flag days, say changes that impact langtools+jdk
>>> or hotspot+jdk and these usually cause a bit of disruption until the
>>> changes get into a promoted build and into the import JDK that people
>>> use.  As to why people do partial builds then it's probably a mix of
>>> issues including disk space, time to do the first build, and of course
>>> it's just the way that we've always done it.
>>
>>
>> In build-infra, there is currently a "somewhat partial  build" feature
>> that is implemented like this:
>> 1) You check out a "master forest", containing all repos. You only need
>> to do this checkout once, and you are not required to pull/update it
>> (unless you want to keep up with changes -- like on a flag day).
>>
>> 2) For subsequent work, you can clone just a single repo (let's call it
>> "my_hotspot"), and doing some configure magic (currently a bit
>> complicated, but that can be rewritten to be simpler) which will point
>> to the "master forest" but replace the hotspot repo in the master forest
>> with "my_hotspot"..
>>
>> 3) The first time you build "my_hotspot", it will rebuild the whole
>> forest (but with hotspot replaced with my_hotspot). However, if you
>> never pull/update the master forest, this will be a one-time event.
>> Subsequent builds will see that nothing has changed in the master
>> forest, and only changes in my_hotspot will be recompiled.
>>
>> The pros with this solution is:
>> 1) help with flag days -- just pull/update the master forest and rebuild.
>> 2) you can always build the complete product.
>> 3) you save some disk space by only cloning the complete forest once.
>>
>> However:
>> 1) you will still need disk space for build output for the complete
>> forest, for every single repo.
>> 2) you will still need to recompile the whole forest for each new single
>> repo you clone.
>>
>> Do you think this will make the solution not good enough?
>>
>> I think it is possible and not too hard to change this solution
>> slightly, so you would only need to compile the master forest once, and
>> then new single repos can use that build output as well as using the
>> source code from the master forest. Though proper dependency checking
>> might be harder, and there will be semantic issues about the meaning of
>> things like "make clean" that can be surprising. Is this something you
>> think is worth pursuing?
>>
>> To completely eliminate compiling the whole product at least once seems
>> much harder. I can't really see a good way of using "import JDKs" the
>> way they've been used in the old build system. I imagine one possibility
>> is to prepare a tar ball, including a complete cloned forest (for a
>> certain point in time) and a build directory with build artifacts
>> corresponding to that source code and your platform(s) of choice, and
>> have people download that as starting point. It seems like a lot of
>> work, though. Do you think that is needed to get the new build system
>> accepted?
>>
>> /Magnus