RFR: JDK-8146403: Windows build can be faster

[Magnus Ihse Bursie]

On 2016-01-14 11:43, Erik Joelsson wrote:
> Hello,
>
> Thinking more about the new bootstrap logic, I feel that I don't like 
> the solution. Here is a new patch where I only changed that part. I 
> reverted to using the -include mechanism to trigger generation of 
> module-deps.gmk, but still in a separate makefile that is only 
> included by Main.gmk.
>
> Webrev: http://cr.openjdk.java.net/~erikj/8146403/webrev.03/

It looks like this version is just like the original, but you have moved 
some code from Modules.gmk to GenerateModuleDeps.gmk, but since both of 
them is included in Main.gmk, there will be no functional changes, just 
a matter of code cleanup. Or am I missing something?

I'm not saying it's bad, I just understand how and if this resolves what 
you set out to solve in your first version.

/Magnus

>
> /Erik
>
> On 2016-01-13 14:44, Erik Joelsson wrote:
>> Hello,
>>
>> New webrev: http://cr.openjdk.java.net/~erikj/8146403/webrev.02/
>>
>> Fixed the below comments. Also adjusted some more gensrc log output 
>> to make it more uniform, even if it doesn't exactly address Windows 
>> performance.
>>
>> I also fixed a bug in the bootstrap make logic. The module-deps.gmk 
>> wasn't correctly generated in all cases. Typically when building more 
>> than one conf at a time or when doing a compare build. I added 
>> explicit calls to GenerateModuleDeps.gmk to fix this.
>>
>> /Erik
>>
>> On 2016-01-13 12:48, Magnus Ihse Bursie wrote:
>>> Hi,
>>>
>>> In InitSupport.gmk, please restore the comment:
>>>  # Only do this if make has not been restarted, and if we do not 
>>> force it.
>>>
>>> In jdk/make/gensrc/GensrcExceptions.gmk:
>>> The old construct resulted in the output "Generating exceptions 
>>> classes" (but a bit irregularly), could you please re-add it as a 
>>> LogInfo?
>>>
>>> In make/CompileTools.gmk, the copyright header says 2014, which is 
>>> at least one year too early. :-) (In fact, all modified files should 
>>> really get bumped to 2016).
>>>
>>> Apart from this, it looks good to me.
>>>
>>> /Magnus
>>>
>>> On 2016-01-05 11:58, Erik Joelsson wrote:
>>>> Hello,
>>>>
>>>> During the hotspot makefile conversion, we have been reminded of 
>>>> inefficiencies when running make in Cygwin. We still have a pretty 
>>>> severe performance regression in the new hotspot build compared to 
>>>> the old on Windows in certain situations, my laptop being one such 
>>>> situation. A recent comparison there between just the old and new 
>>>> hotspot build:
>>>>
>>>> release new 00:04:30
>>>> release old 00:03:10
>>>> fastdebug new 00:04:59
>>>> fastdebug old 00:03:37
>>>>
>>>> Much of the extra time is spent spawning various shell processes 
>>>> for book keeping, like saving failure logs, creating header 
>>>> dependency files etc.
>>>>
>>>> It also takes a very long time to do a "do nothing" rebuild when 
>>>> nothing has changed. On my laptop, repeating "make jimages" often 
>>>> takes as long as 40 seconds to figure out that nothing needs to be 
>>>> rebuilt. In this case there are several culprits.
>>>>
>>>> I have been working on improvements in these areas to reduce the 
>>>> overhead. The "do nothing" rebuild of jimages is down to around 25 
>>>> seconds. A full images build is around 1-2 minutes faster from 24 
>>>> to 22 minutes, but fluctuates quite a bit. The new hotspot build is 
>>>> also improved:
>>>>
>>>> release: 3:44
>>>> fastdebug: 4:02
>>>>
>>>> Here is a list of the kinds of changes I've made:
>>>>
>>>> * Rewrote logger.sh to use a different construct. It drastically 
>>>> reduces the number of bash processes being spawned. Basically you 
>>>> can pipe like this: "> >(tee logfile) 2> >(tee logfile >&2)". I 
>>>> also managed to reduce the extra sed/grep commands needed for the 
>>>> header dependencies even more. A side effect of this is that the 
>>>> log files for each native compilation unit are always saved instead 
>>>> of being deleted on successful compilation. I see no issue with this.
>>>>
>>>> * Changed all recipes that contain echo for logging to instead use 
>>>> new LogInfo macro, which in turn calls $(info ) if appropriate.
>>>>
>>>> * Changed all recipes that contain mkdir to instead use MakeDir 
>>>> macro, which only executes mkdir if the directory doesn't exist.
>>>>
>>>> * In HotspotWrapper.gmk there is an optimization to avoid calling 
>>>> the hotspot makefiles at all if nothing has changed. This runs a 
>>>> find over the whole hotspot repository. I reduced this to only run 
>>>> over src and make sub dirs to avoid the pretty large test dir 
>>>> (since test/closed has grown quite a bit).
>>>>
>>>> * Split Tools.gmk into CompileTools.gmk and Tools.gmk, to avoid 
>>>> having the buildtools compilation being reevaluated by all 
>>>> makefiles needing the tools definitions.
>>>>
>>>> * Split Modules.gmk to avoid having the module deps generation 
>>>> being reevaluated multiple times. Made the new 
>>>> GenerateModuleDeps.gmk an explicit call from Init.gmk.
>>>>
>>>> Since these improvements affect much more than just the new hotspot 
>>>> build, I intend to push this to JDK 9 directly.
>>>>
>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8146403
>>>> Webrev: http://cr.openjdk.java.net/~erikj/8146403/webrev.01/
>>>>
>>>> /Erik
>>>
>>
>