RFR: JDK-8220383: Incremental build is broken and inefficient

[Tim Bell]

Erik:

Looks good to me.

Tim

> Our incremental build performance has degraded a lot recently. On
> Windows it's especially bad (as usual). There are multiple issues. Some
> cases of running way too many find/grep during rule setup. Some files
> get rebuilt on every incremental build. With all these changes, the
> incremental build with no changes of "make product-bundles test-bundles"
> on a Windows machine I was testing went from 4m to 24s. Here is a list
> of the fixes:
>
> We have a rather inefficient pattern where a makefile is responsible for
> two top level targets, the general build of something and then the
> copying of that something into images. The drawback of this pattern is
> that all the rules in the makefile get evaluated twice, and this is
> potentially expensive. My workaround for this is to enclose the
> expensive parts in a conditional that checks what target the makefile
> was called with. This pattern is already in use in at least Bundles.gmk.
>
> In CompileJavaModules.gmk, if IMPORT_MODULES_CLASSES isn't set, wildcard
> will try to look for /$(MODULE), which can sometimes be very costly and
> freeze the system for 10-20s (likely due to going to check on the network).
>
> Images.gmk, on Windows, the CDS archive was always being rebuilt due to
> the rule targeting the wrong file.
>
> NativeCompilation.gmk has two fixes. Reduce the number of $(shell find
> ...) calls by doing just one and then ordering the files found. The
> second fix is more intricate. The adding of the IMPORT_LIBRARY to the
> list of TARGETS was missing. This was causing trouble in combination
> with a test lib, libHasNoEntryPoint.c, which does not produce a link.lib
> file. When building incrementally, the makefile cannot find the
> HasNoEntryPoint.lib file and so deletes HasNoEntryPoint.dll to force a
> rebuild. By adding the IMPORT_LIBRARY to the targets, the
> HasNoEntryPoint.lib will at least be touched. I also had to remove the
> safety check that a .lib file was really created since it apparently
> isn't always created.
>
> In ProcessMarkdown.gmk, the $(shell grep ) expression was called for
> every markdown file, even if no rebuilding was needed.
>
> TestFilesCompilation.gmk called SetupNativeCompilation for each test
> .c/cpp file, which would in turn call find on the src dir where that
> file was located. I changed the call to use the EXTRA_FILES parameter
> instead of SRC to avoid the extra calls to find.
>
> In CompileJvm.gmk I added a call to FillCacheFind so we can reuse the
> set of hotspot sources since they are used more than once.
>
> I have run a comparison build of our main platforms with the patch.
>
> There are certainly more improvements possible, but I chose to limit
> myself to the most obviously broken parts here.
>
> Webrev: http://cr.openjdk.java.net/~erikj/8220383/webrev.01/index.html
>
> Bug: https://bugs.openjdk.java.net/browse/JDK-8220383
>
> /Erik