RFR: JDK-8146403: Windows build can be faster

Thu Jan 14 11:07:14 UTC 2016

On 2016-01-14 12:01, Erik Joelsson wrote:
>
>
> On 2016-01-14 11:53, Magnus Ihse Bursie wrote:
>> On 2016-01-14 11:43, Erik Joelsson wrote:
>>> Hello,
>>>
>>> Thinking more about the new bootstrap logic, I feel that I don't 
>>> like the solution. Here is a new patch where I only changed that 
>>> part. I reverted to using the -include mechanism to trigger 
>>> generation of module-deps.gmk, but still in a separate makefile that 
>>> is only included by Main.gmk.
>>>
>>> Webrev: http://cr.openjdk.java.net/~erikj/8146403/webrev.03/
>>
>> It looks like this version is just like the original, but you have 
>> moved some code from Modules.gmk to GenerateModuleDeps.gmk, but since 
>> both of them is included in Main.gmk, there will be no functional 
>> changes, just a matter of code cleanup. Or am I missing something?
>>
>> I'm not saying it's bad, I just understand how and if this resolves 
>> what you set out to solve in your first version.
>>
> Modules.gmk is included in many places. Each such instance would have 
> to evaluate if module-deps.gmk needed to be recreated or not, and in 
> turn run a find over the source files for the generator tool. This is 
> now only done in Main.gmk.
Alright, I didn't see those includes. Sounds good then. Ship it! :)

/Magnus

>
> /Erik
>> /Magnus
>>
>>>
>>> /Erik
>>>
>>> On 2016-01-13 14:44, Erik Joelsson wrote:
>>>> Hello,
>>>>
>>>> New webrev: http://cr.openjdk.java.net/~erikj/8146403/webrev.02/
>>>>
>>>> Fixed the below comments. Also adjusted some more gensrc log output 
>>>> to make it more uniform, even if it doesn't exactly address Windows 
>>>> performance.
>>>>
>>>> I also fixed a bug in the bootstrap make logic. The module-deps.gmk 
>>>> wasn't correctly generated in all cases. Typically when building 
>>>> more than one conf at a time or when doing a compare build. I added 
>>>> explicit calls to GenerateModuleDeps.gmk to fix this.
>>>>
>>>> /Erik
>>>>
>>>> On 2016-01-13 12:48, Magnus Ihse Bursie wrote:
>>>>> Hi,
>>>>>
>>>>> In InitSupport.gmk, please restore the comment:
>>>>>  # Only do this if make has not been restarted, and if we do not 
>>>>> force it.
>>>>>
>>>>> In jdk/make/gensrc/GensrcExceptions.gmk:
>>>>> The old construct resulted in the output "Generating exceptions 
>>>>> classes" (but a bit irregularly), could you please re-add it as a 
>>>>> LogInfo?
>>>>>
>>>>> In make/CompileTools.gmk, the copyright header says 2014, which is 
>>>>> at least one year too early. :-) (In fact, all modified files 
>>>>> should really get bumped to 2016).
>>>>>
>>>>> Apart from this, it looks good to me.
>>>>>
>>>>> /Magnus
>>>>>
>>>>> On 2016-01-05 11:58, Erik Joelsson wrote:
>>>>>> Hello,
>>>>>>
>>>>>> During the hotspot makefile conversion, we have been reminded of 
>>>>>> inefficiencies when running make in Cygwin. We still have a 
>>>>>> pretty severe performance regression in the new hotspot build 
>>>>>> compared to the old on Windows in certain situations, my laptop 
>>>>>> being one such situation. A recent comparison there between just 
>>>>>> the old and new hotspot build:
>>>>>>
>>>>>> release new 00:04:30
>>>>>> release old 00:03:10
>>>>>> fastdebug new 00:04:59
>>>>>> fastdebug old 00:03:37
>>>>>>
>>>>>> Much of the extra time is spent spawning various shell processes 
>>>>>> for book keeping, like saving failure logs, creating header 
>>>>>> dependency files etc.
>>>>>>
>>>>>> It also takes a very long time to do a "do nothing" rebuild when 
>>>>>> nothing has changed. On my laptop, repeating "make jimages" often 
>>>>>> takes as long as 40 seconds to figure out that nothing needs to 
>>>>>> be rebuilt. In this case there are several culprits.
>>>>>>
>>>>>> I have been working on improvements in these areas to reduce the 
>>>>>> overhead. The "do nothing" rebuild of jimages is down to around 
>>>>>> 25 seconds. A full images build is around 1-2 minutes faster from 
>>>>>> 24 to 22 minutes, but fluctuates quite a bit. The new hotspot 
>>>>>> build is also improved:
>>>>>>
>>>>>> release: 3:44
>>>>>> fastdebug: 4:02
>>>>>>
>>>>>> Here is a list of the kinds of changes I've made:
>>>>>>
>>>>>> * Rewrote logger.sh to use a different construct. It drastically 
>>>>>> reduces the number of bash processes being spawned. Basically you 
>>>>>> can pipe like this: "> >(tee logfile) 2> >(tee logfile >&2)". I 
>>>>>> also managed to reduce the extra sed/grep commands needed for the 
>>>>>> header dependencies even more. A side effect of this is that the 
>>>>>> log files for each native compilation unit are always saved 
>>>>>> instead of being deleted on successful compilation. I see no 
>>>>>> issue with this.
>>>>>>
>>>>>> * Changed all recipes that contain echo for logging to instead 
>>>>>> use new LogInfo macro, which in turn calls $(info ) if appropriate.
>>>>>>
>>>>>> * Changed all recipes that contain mkdir to instead use MakeDir 
>>>>>> macro, which only executes mkdir if the directory doesn't exist.
>>>>>>
>>>>>> * In HotspotWrapper.gmk there is an optimization to avoid calling 
>>>>>> the hotspot makefiles at all if nothing has changed. This runs a 
>>>>>> find over the whole hotspot repository. I reduced this to only 
>>>>>> run over src and make sub dirs to avoid the pretty large test dir 
>>>>>> (since test/closed has grown quite a bit).
>>>>>>
>>>>>> * Split Tools.gmk into CompileTools.gmk and Tools.gmk, to avoid 
>>>>>> having the buildtools compilation being reevaluated by all 
>>>>>> makefiles needing the tools definitions.
>>>>>>
>>>>>> * Split Modules.gmk to avoid having the module deps generation 
>>>>>> being reevaluated multiple times. Made the new 
>>>>>> GenerateModuleDeps.gmk an explicit call from Init.gmk.
>>>>>>
>>>>>> Since these improvements affect much more than just the new 
>>>>>> hotspot build, I intend to push this to JDK 9 directly.
>>>>>>
>>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8146403
>>>>>> Webrev: http://cr.openjdk.java.net/~erikj/8146403/webrev.01/
>>>>>>
>>>>>> /Erik
>>>>>
>>>>
>>>
>>
>