RFR: JDK-8146403: Windows build can be faster

[Erik Joelsson]

On 2016-01-14 11:53, Magnus Ihse Bursie wrote:
> On 2016-01-14 11:43, Erik Joelsson wrote:
>> Hello,
>>
>> Thinking more about the new bootstrap logic, I feel that I don't like 
>> the solution. Here is a new patch where I only changed that part. I 
>> reverted to using the -include mechanism to trigger generation of 
>> module-deps.gmk, but still in a separate makefile that is only 
>> included by Main.gmk.
>>
>> Webrev: http://cr.openjdk.java.net/~erikj/8146403/webrev.03/
>
> It looks like this version is just like the original, but you have 
> moved some code from Modules.gmk to GenerateModuleDeps.gmk, but since 
> both of them is included in Main.gmk, there will be no functional 
> changes, just a matter of code cleanup. Or am I missing something?
>
> I'm not saying it's bad, I just understand how and if this resolves 
> what you set out to solve in your first version.
>
Modules.gmk is included in many places. Each such instance would have to 
evaluate if module-deps.gmk needed to be recreated or not, and in turn 
run a find over the source files for the generator tool. This is now 
only done in Main.gmk.

/Erik
> /Magnus
>
>>
>> /Erik
>>
>> On 2016-01-13 14:44, Erik Joelsson wrote:
>>> Hello,
>>>
>>> New webrev: http://cr.openjdk.java.net/~erikj/8146403/webrev.02/
>>>
>>> Fixed the below comments. Also adjusted some more gensrc log output 
>>> to make it more uniform, even if it doesn't exactly address Windows 
>>> performance.
>>>
>>> I also fixed a bug in the bootstrap make logic. The module-deps.gmk 
>>> wasn't correctly generated in all cases. Typically when building 
>>> more than one conf at a time or when doing a compare build. I added 
>>> explicit calls to GenerateModuleDeps.gmk to fix this.
>>>
>>> /Erik
>>>
>>> On 2016-01-13 12:48, Magnus Ihse Bursie wrote:
>>>> Hi,
>>>>
>>>> In InitSupport.gmk, please restore the comment:
>>>>  # Only do this if make has not been restarted, and if we do not 
>>>> force it.
>>>>
>>>> In jdk/make/gensrc/GensrcExceptions.gmk:
>>>> The old construct resulted in the output "Generating exceptions 
>>>> classes" (but a bit irregularly), could you please re-add it as a 
>>>> LogInfo?
>>>>
>>>> In make/CompileTools.gmk, the copyright header says 2014, which is 
>>>> at least one year too early. :-) (In fact, all modified files 
>>>> should really get bumped to 2016).
>>>>
>>>> Apart from this, it looks good to me.
>>>>
>>>> /Magnus
>>>>
>>>> On 2016-01-05 11:58, Erik Joelsson wrote:
>>>>> Hello,
>>>>>
>>>>> During the hotspot makefile conversion, we have been reminded of 
>>>>> inefficiencies when running make in Cygwin. We still have a pretty 
>>>>> severe performance regression in the new hotspot build compared to 
>>>>> the old on Windows in certain situations, my laptop being one such 
>>>>> situation. A recent comparison there between just the old and new 
>>>>> hotspot build:
>>>>>
>>>>> release new 00:04:30
>>>>> release old 00:03:10
>>>>> fastdebug new 00:04:59
>>>>> fastdebug old 00:03:37
>>>>>
>>>>> Much of the extra time is spent spawning various shell processes 
>>>>> for book keeping, like saving failure logs, creating header 
>>>>> dependency files etc.
>>>>>
>>>>> It also takes a very long time to do a "do nothing" rebuild when 
>>>>> nothing has changed. On my laptop, repeating "make jimages" often 
>>>>> takes as long as 40 seconds to figure out that nothing needs to be 
>>>>> rebuilt. In this case there are several culprits.
>>>>>
>>>>> I have been working on improvements in these areas to reduce the 
>>>>> overhead. The "do nothing" rebuild of jimages is down to around 25 
>>>>> seconds. A full images build is around 1-2 minutes faster from 24 
>>>>> to 22 minutes, but fluctuates quite a bit. The new hotspot build 
>>>>> is also improved:
>>>>>
>>>>> release: 3:44
>>>>> fastdebug: 4:02
>>>>>
>>>>> Here is a list of the kinds of changes I've made:
>>>>>
>>>>> * Rewrote logger.sh to use a different construct. It drastically 
>>>>> reduces the number of bash processes being spawned. Basically you 
>>>>> can pipe like this: "> >(tee logfile) 2> >(tee logfile >&2)". I 
>>>>> also managed to reduce the extra sed/grep commands needed for the 
>>>>> header dependencies even more. A side effect of this is that the 
>>>>> log files for each native compilation unit are always saved 
>>>>> instead of being deleted on successful compilation. I see no issue 
>>>>> with this.
>>>>>
>>>>> * Changed all recipes that contain echo for logging to instead use 
>>>>> new LogInfo macro, which in turn calls $(info ) if appropriate.
>>>>>
>>>>> * Changed all recipes that contain mkdir to instead use MakeDir 
>>>>> macro, which only executes mkdir if the directory doesn't exist.
>>>>>
>>>>> * In HotspotWrapper.gmk there is an optimization to avoid calling 
>>>>> the hotspot makefiles at all if nothing has changed. This runs a 
>>>>> find over the whole hotspot repository. I reduced this to only run 
>>>>> over src and make sub dirs to avoid the pretty large test dir 
>>>>> (since test/closed has grown quite a bit).
>>>>>
>>>>> * Split Tools.gmk into CompileTools.gmk and Tools.gmk, to avoid 
>>>>> having the buildtools compilation being reevaluated by all 
>>>>> makefiles needing the tools definitions.
>>>>>
>>>>> * Split Modules.gmk to avoid having the module deps generation 
>>>>> being reevaluated multiple times. Made the new 
>>>>> GenerateModuleDeps.gmk an explicit call from Init.gmk.
>>>>>
>>>>> Since these improvements affect much more than just the new 
>>>>> hotspot build, I intend to push this to JDK 9 directly.
>>>>>
>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8146403
>>>>> Webrev: http://cr.openjdk.java.net/~erikj/8146403/webrev.01/
>>>>>
>>>>> /Erik
>>>>
>>>
>>
>