RFR: JDK-8056053: Disable HOTSPOT_BUILD_JOBS when building with configure

[David Holmes]

On 26/08/2014 10:38 PM, Erik Joelsson wrote:
> Hello David,
>
> I realize my description was not very clear and will try to make a
> better one.

Thanks for the detailed explanation. Changes seem good.

David

> It used to be (before make 3.78), that recursive make calls and the -j
> flag were tricky to use together. The Hotspot makefiles solved this by
> asking people not to use -j, and instead set the variable
> HOTSPOT_BUILD_JOBS. The calls to the leaf make invocations would then
> set -j$(HOTSPOT_BUILD_JOBS) so that the bulk of the work would be run in
> parallel, but one makefile at a time.
>
> Modern versions of gnu make has a solution to this problem called the
> Job Server. (Read more about it here:
> http://mad-scientist.net/make/jobserver.html) In short it enables
> multiple make processes to share a single pool of job tokens through a
> pipe, making sure that concurrency is kept within the limitations set by
> -j to the root process. Note that make never sends the -j flag on to sub
> invocations, but instead does a secret transfer of the Job Server pipe.
>
> In the "new" jdk9 build, I introduced using the Job Server across the
> whole build. Unfortunately, this didn't work well for Hotspot because as
> soon as one sub make gets an explicit -j flag, or is invoked from a
> recipe line without either a + sign or explicit use of the $(MAKE)
> variable, that subtree is outside the job server sharing pool and will
> govern its own concurrency. The result of this is that while the rest of
> the build is sharing JOBS number of parallel jobs, vm.make will execute
> HOTSPOT_BUILD_NUMBER number of parallel jobs on its own.
>
> The reason .NOTPARALLEL wasn't need before is that until this patch,
> HotspotWrapper.gmk and the user building Hotspot alone has always run
> make with -j1. In this patch, I removed that -j1 from HotspotWrapper.gmk
> so that the Job Server would get propagated into the Hotspot build. To
> protect the makefiles that aren't able to run in parallel, I had to add
> .NOTPARALLEL: in a couple of places.
>
> /Erik
>
> On 2014-08-26 14:09, David Holmes wrote:
>> Hi Erik,
>>
>> On 26/08/2014 8:53 PM, Erik Joelsson wrote:
>>> Hello,
>>>
>>> Please review this proposed fix for the Hotspot build.
>>>
>>> In the new jdk9 build, we utilize the gnu make job server, which
>>> automatically makes sure the -j flag gets propagated and shared between
>>> all recursive make calls. In the hotspot build, this gets overridden by
>>> the HOTSPOT_BUILD_JOBS variable. Configure estimates a reasonable number
>>> of parallel make jobs into the JOBS variable, which gets propagated to
>>> the HOTSPOT_BUILD_JOBS variable. This used to work well enough, but in
>>> the new build, the hotspot build is happening concurrently with other
>>> parts of the build and the consequence is that the hotspot build gets
>>> JOBS number of jobs and the rest of the build also gets JOBS number of
>>> jobs, all of which are used at the same time. We would like the whole
>>> build to share in the same job pool.
>>>
>>> To fix this, the setting of -j$(HOTSPOT_BUILD_JOBS) needs to be made
>>> conditional and we need to add .NOTPARALLEL: to a number of makefiles in
>>> hotspot that currently can't handle being executed in parallel. Lastly,
>>> the + sign must be added first to recipe lines that call make
>>> recursively but are not explicitly using the MAKE variable directly. The
>>> result will be that the active -j flag in the root makefiles will just
>>> automatically propagate down to the hotspot makefiles.
>>
>> I don't know how make handles -j propagation but from your description
>> it sounds like we will still generate too much concurrency as the
>> submakes will be run with the same -j value as the top-level. ??
>>
>> I'm also unclear how we used to pass HOTSPOT_BUILD_JOBS and didn't
>> need .NOTPARALLEL, but now we pass -j from the top we do need
>> .NOTPARALLEL ??
>>
>> Thanks,
>> David
>>
>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8056053
>>> Webrev: http://cr.openjdk.java.net/~erikj/8056053/webrev.01/
>>>
>>> /Erik
>