The future of partial builds

[Anthony Petrov]

+1

--
best regards,
Anthony

On 9/10/2012 6:52 PM, Weijun Wang wrote:
> Sorry I was not clear, by "clone the repo(s)", I mean either only the 
> jdk repo, or plus the jdk/*/closed ones. I almost never clone other 
> repos (langtools, hotspot, ...).
> 
> -Max
> 
> On 09/10/2012 10:48 PM, Weijun Wang wrote:
>> It's not uncommon that I just clone the repo(s), do a quick build, run
>> several tests and abandon it, for example, if I want to try my code
>> changes on a different platform (from my daily work machine). Therefore
>> I find the jdk-only build very useful.
>>
>> -Max
>>
>> On 09/10/2012 10:20 PM, Magnus Ihse Bursie wrote:
>>> On 2012-09-10 14:13, Alan Bateman wrote:
>>>>
>>>> I think this is a great topic to discuss.
>>>>
>>>> At least within Oracle then I think the majority of people do partial
>>>> builds in their local environment. When I say "partial build" then I
>>>> mean they build a subset of repositories, not all of them. So folks
>>>> working in the jdk repository tend to just build that repository with
>>>> everything else coming from whatever import JDK they are using.
>>>> Partial builds are of course very fragile and frequently people have
>>>> problems because they aren't using a matching import JDK. Also
>>>> periodically we have flag days, say changes that impact langtools+jdk
>>>> or hotspot+jdk and these usually cause a bit of disruption until the
>>>> changes get into a promoted build and into the import JDK that people
>>>> use.  As to why people do partial builds then it's probably a mix of
>>>> issues including disk space, time to do the first build, and of course
>>>> it's just the way that we've always done it.
>>>
>>>
>>> In build-infra, there is currently a "somewhat partial  build" feature
>>> that is implemented like this:
>>> 1) You check out a "master forest", containing all repos. You only need
>>> to do this checkout once, and you are not required to pull/update it
>>> (unless you want to keep up with changes -- like on a flag day).
>>>
>>> 2) For subsequent work, you can clone just a single repo (let's call it
>>> "my_hotspot"), and doing some configure magic (currently a bit
>>> complicated, but that can be rewritten to be simpler) which will point
>>> to the "master forest" but replace the hotspot repo in the master forest
>>> with "my_hotspot"..
>>>
>>> 3) The first time you build "my_hotspot", it will rebuild the whole
>>> forest (but with hotspot replaced with my_hotspot). However, if you
>>> never pull/update the master forest, this will be a one-time event.
>>> Subsequent builds will see that nothing has changed in the master
>>> forest, and only changes in my_hotspot will be recompiled.
>>>
>>> The pros with this solution is:
>>> 1) help with flag days -- just pull/update the master forest and 
>>> rebuild.
>>> 2) you can always build the complete product.
>>> 3) you save some disk space by only cloning the complete forest once.
>>>
>>> However:
>>> 1) you will still need disk space for build output for the complete
>>> forest, for every single repo.
>>> 2) you will still need to recompile the whole forest for each new single
>>> repo you clone.
>>>
>>> Do you think this will make the solution not good enough?
>>>
>>> I think it is possible and not too hard to change this solution
>>> slightly, so you would only need to compile the master forest once, and
>>> then new single repos can use that build output as well as using the
>>> source code from the master forest. Though proper dependency checking
>>> might be harder, and there will be semantic issues about the meaning of
>>> things like "make clean" that can be surprising. Is this something you
>>> think is worth pursuing?
>>>
>>> To completely eliminate compiling the whole product at least once seems
>>> much harder. I can't really see a good way of using "import JDKs" the
>>> way they've been used in the old build system. I imagine one possibility
>>> is to prepare a tar ball, including a complete cloned forest (for a
>>> certain point in time) and a build directory with build artifacts
>>> corresponding to that source code and your platform(s) of choice, and
>>> have people download that as starting point. It seems like a lot of
>>> work, though. Do you think that is needed to get the new build system
>>> accepted?
>>>
>>> /Magnus