RFR: JDK-8146403: Windows build can be faster

[Erik Joelsson]

Hello,

Thinking more about the new bootstrap logic, I feel that I don't like 
the solution. Here is a new patch where I only changed that part. I 
reverted to using the -include mechanism to trigger generation of 
module-deps.gmk, but still in a separate makefile that is only included 
by Main.gmk.

Webrev: http://cr.openjdk.java.net/~erikj/8146403/webrev.03/

/Erik

On 2016-01-13 14:44, Erik Joelsson wrote:
> Hello,
>
> New webrev: http://cr.openjdk.java.net/~erikj/8146403/webrev.02/
>
> Fixed the below comments. Also adjusted some more gensrc log output to 
> make it more uniform, even if it doesn't exactly address Windows 
> performance.
>
> I also fixed a bug in the bootstrap make logic. The module-deps.gmk 
> wasn't correctly generated in all cases. Typically when building more 
> than one conf at a time or when doing a compare build. I added 
> explicit calls to GenerateModuleDeps.gmk to fix this.
>
> /Erik
>
> On 2016-01-13 12:48, Magnus Ihse Bursie wrote:
>> Hi,
>>
>> In InitSupport.gmk, please restore the comment:
>>  # Only do this if make has not been restarted, and if we do not 
>> force it.
>>
>> In jdk/make/gensrc/GensrcExceptions.gmk:
>> The old construct resulted in the output "Generating exceptions 
>> classes" (but a bit irregularly), could you please re-add it as a 
>> LogInfo?
>>
>> In make/CompileTools.gmk, the copyright header says 2014, which is at 
>> least one year too early. :-) (In fact, all modified files should 
>> really get bumped to 2016).
>>
>> Apart from this, it looks good to me.
>>
>> /Magnus
>>
>> On 2016-01-05 11:58, Erik Joelsson wrote:
>>> Hello,
>>>
>>> During the hotspot makefile conversion, we have been reminded of 
>>> inefficiencies when running make in Cygwin. We still have a pretty 
>>> severe performance regression in the new hotspot build compared to 
>>> the old on Windows in certain situations, my laptop being one such 
>>> situation. A recent comparison there between just the old and new 
>>> hotspot build:
>>>
>>> release new 00:04:30
>>> release old 00:03:10
>>> fastdebug new 00:04:59
>>> fastdebug old 00:03:37
>>>
>>> Much of the extra time is spent spawning various shell processes for 
>>> book keeping, like saving failure logs, creating header dependency 
>>> files etc.
>>>
>>> It also takes a very long time to do a "do nothing" rebuild when 
>>> nothing has changed. On my laptop, repeating "make jimages" often 
>>> takes as long as 40 seconds to figure out that nothing needs to be 
>>> rebuilt. In this case there are several culprits.
>>>
>>> I have been working on improvements in these areas to reduce the 
>>> overhead. The "do nothing" rebuild of jimages is down to around 25 
>>> seconds. A full images build is around 1-2 minutes faster from 24 to 
>>> 22 minutes, but fluctuates quite a bit. The new hotspot build is 
>>> also improved:
>>>
>>> release: 3:44
>>> fastdebug: 4:02
>>>
>>> Here is a list of the kinds of changes I've made:
>>>
>>> * Rewrote logger.sh to use a different construct. It drastically 
>>> reduces the number of bash processes being spawned. Basically you 
>>> can pipe like this: "> >(tee logfile) 2> >(tee logfile >&2)". I also 
>>> managed to reduce the extra sed/grep commands needed for the header 
>>> dependencies even more. A side effect of this is that the log files 
>>> for each native compilation unit are always saved instead of being 
>>> deleted on successful compilation. I see no issue with this.
>>>
>>> * Changed all recipes that contain echo for logging to instead use 
>>> new LogInfo macro, which in turn calls $(info ) if appropriate.
>>>
>>> * Changed all recipes that contain mkdir to instead use MakeDir 
>>> macro, which only executes mkdir if the directory doesn't exist.
>>>
>>> * In HotspotWrapper.gmk there is an optimization to avoid calling 
>>> the hotspot makefiles at all if nothing has changed. This runs a 
>>> find over the whole hotspot repository. I reduced this to only run 
>>> over src and make sub dirs to avoid the pretty large test dir (since 
>>> test/closed has grown quite a bit).
>>>
>>> * Split Tools.gmk into CompileTools.gmk and Tools.gmk, to avoid 
>>> having the buildtools compilation being reevaluated by all makefiles 
>>> needing the tools definitions.
>>>
>>> * Split Modules.gmk to avoid having the module deps generation being 
>>> reevaluated multiple times. Made the new GenerateModuleDeps.gmk an 
>>> explicit call from Init.gmk.
>>>
>>> Since these improvements affect much more than just the new hotspot 
>>> build, I intend to push this to JDK 9 directly.
>>>
>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8146403
>>> Webrev: http://cr.openjdk.java.net/~erikj/8146403/webrev.01/
>>>
>>> /Erik
>>
>