RFR: JDK-8056053: Disable HOTSPOT_BUILD_JOBS when building with configure

[Erik Joelsson]

Hello David,

I realize my description was not very clear and will try to make a 
better one.

It used to be (before make 3.78), that recursive make calls and the -j 
flag were tricky to use together. The Hotspot makefiles solved this by 
asking people not to use -j, and instead set the variable 
HOTSPOT_BUILD_JOBS. The calls to the leaf make invocations would then 
set -j$(HOTSPOT_BUILD_JOBS) so that the bulk of the work would be run in 
parallel, but one makefile at a time.

Modern versions of gnu make has a solution to this problem called the 
Job Server. (Read more about it here: 
http://mad-scientist.net/make/jobserver.html) In short it enables 
multiple make processes to share a single pool of job tokens through a 
pipe, making sure that concurrency is kept within the limitations set by 
-j to the root process. Note that make never sends the -j flag on to sub 
invocations, but instead does a secret transfer of the Job Server pipe.

In the "new" jdk9 build, I introduced using the Job Server across the 
whole build. Unfortunately, this didn't work well for Hotspot because as 
soon as one sub make gets an explicit -j flag, or is invoked from a 
recipe line without either a + sign or explicit use of the $(MAKE) 
variable, that subtree is outside the job server sharing pool and will 
govern its own concurrency. The result of this is that while the rest of 
the build is sharing JOBS number of parallel jobs, vm.make will execute 
HOTSPOT_BUILD_NUMBER number of parallel jobs on its own.

The reason .NOTPARALLEL wasn't need before is that until this patch, 
HotspotWrapper.gmk and the user building Hotspot alone has always run 
make with -j1. In this patch, I removed that -j1 from HotspotWrapper.gmk 
so that the Job Server would get propagated into the Hotspot build. To 
protect the makefiles that aren't able to run in parallel, I had to add 
.NOTPARALLEL: in a couple of places.

/Erik

On 2014-08-26 14:09, David Holmes wrote:
> Hi Erik,
>
> On 26/08/2014 8:53 PM, Erik Joelsson wrote:
>> Hello,
>>
>> Please review this proposed fix for the Hotspot build.
>>
>> In the new jdk9 build, we utilize the gnu make job server, which
>> automatically makes sure the -j flag gets propagated and shared between
>> all recursive make calls. In the hotspot build, this gets overridden by
>> the HOTSPOT_BUILD_JOBS variable. Configure estimates a reasonable number
>> of parallel make jobs into the JOBS variable, which gets propagated to
>> the HOTSPOT_BUILD_JOBS variable. This used to work well enough, but in
>> the new build, the hotspot build is happening concurrently with other
>> parts of the build and the consequence is that the hotspot build gets
>> JOBS number of jobs and the rest of the build also gets JOBS number of
>> jobs, all of which are used at the same time. We would like the whole
>> build to share in the same job pool.
>>
>> To fix this, the setting of -j$(HOTSPOT_BUILD_JOBS) needs to be made
>> conditional and we need to add .NOTPARALLEL: to a number of makefiles in
>> hotspot that currently can't handle being executed in parallel. Lastly,
>> the + sign must be added first to recipe lines that call make
>> recursively but are not explicitly using the MAKE variable directly. The
>> result will be that the active -j flag in the root makefiles will just
>> automatically propagate down to the hotspot makefiles.
>
> I don't know how make handles -j propagation but from your description 
> it sounds like we will still generate too much concurrency as the 
> submakes will be run with the same -j value as the top-level. ??
>
> I'm also unclear how we used to pass HOTSPOT_BUILD_JOBS and didn't 
> need .NOTPARALLEL, but now we pass -j from the top we do need 
> .NOTPARALLEL ??
>
> Thanks,
> David
>
>> Bug: https://bugs.openjdk.java.net/browse/JDK-8056053
>> Webrev: http://cr.openjdk.java.net/~erikj/8056053/webrev.01/
>>
>> /Erik