RFR: JDK-8150736: Excessive disk space used by build system

[David Holmes]

Hi Erik,

This is a big change that is hard to digest. I assume the actual images 
themselves are unaffected by this change?

An initial nit - why use "product-" in the target names? "images-jdk" 
etc seems perfectly sufficient, and "product" may be a misnomer.

Thanks,
David

On 30/09/2016 8:17 PM, Erik Joelsson wrote:
> Here is a rather large patch which should make life better for most
> people building and developing OpenJDK. The main goal of this patch is
> to reduce unnecessary space used by the build. A side goal also become
> to figure out a better general strategy for handling native debug info
> during the build since that was a large part of the wasted space. These
> are the high level changes:
>
> 1. Removed the dist step in the hotspot build (and the corresponding
>    import step in the jdk build) and linking libjvm directly into
>    support/modules_libs. This removes 2 extra copies of all the hotspot
>    native libraries.
> 2. Made many files in the exploded image ($(OUTPUT_DIR)/jdk) symlinks
>    into support/modules_*. Note that binaries and libraries cannot be
>    symlinks since the image won't run then. Debug symbols and other
>    files seem to work fine though.
> 3. Introduced separate top level targets for building the jdk, jre (and
>    serverjre) images. A developer can run "make product-images-jdk" and
>    skip the others. The test targets now depend on just the jdk image
>    to avoid building extra images that aren't needed for tests.
> 4. Changed SetupNativeCompilation to strip binaries by default unless
>    told not to, and removed the separate strip build step. This lets us
>    get rid of modules_*-stripped.
> 5. Build debug symbols into the OUTPUT_DIR in SetupNativeCompilation
>    (instead of OBJECT_DIR and then copy to OUTPUT_DIR). For normal jdk
>    and hotspot libraries, this means the debug symbols end up directly
>    in support/modules_libs/$MODULE/... and there is only one copy of them.
> 6. If zipping debug symbols, this will also leave the unzipped
>    debugsymbols in OUTPUT_DIR, where we wouldn't do so before. This
>    also means that the exploded image will get symlinks to the unzipped
>    debug symbols regardless of if we are asked to zip them or not. This
>    in turn means that debugging the exploded image will just work. This
>    change is space neutral since we were already keeping a copy of the
>    unzipped debugsymbols in the OBJECT_DIR before, they have just been
>    moved to a more useful place.
> 7. Stop copying debug symbols from test native binaries since we aren't
>    stripping them anyway. Better to just leave debug information in the
>    test binaries.
>
> The reduction in size of course varies a lot depending on OS and
> configuration, but as an example, on my 64bit Linux workstation
> (debug-symbols=zipped) the size of the build directory has gone from
> 5.8GB when building "make images" before to 3.8GB when building "make
> product-images-jdk". The biggest gain is from only building the jdk
> image when only that is needed. A fairer comparison is "make images
> test-image" which has gone from 7.3GB to 5.4GB, much due to not
> duplicating debug information in test binaries. On Solaris the gain is
> generally bigger.
>
> Another nice effect of this is that the exploded image now has debug
> information for all native libraries by default, and that handling of
> dSYM directories on Macosx should now work properly.
>
> The only significant changes detected by comparison builds is that some
> binaries that previously weren't stripped, now are, which they also
> should be.
>
> Bug: https://bugs.openjdk.java.net/browse/JDK-8150736
>
> Webrev: http://cr.openjdk.java.net/~erikj/8150736/webrev.01/
>
> /Erik
>