RFR: JDK-8150736: Excessive disk space used by build system

[David Holmes]

On 3/10/2016 6:22 PM, Erik Joelsson wrote:
> Hello,
>
>
> On 2016-10-01 14:13, David Holmes wrote:
>> Hi Erik,
>>
>> This is a big change that is hard to digest. I assume the actual
>> images themselves are unaffected by this change?
>>
> I understand that and I have been working on this for some time in the
> background. Yes, I have done quite extensive comparisons during this
> work so there should not be any relevant differences in the images.
> There are some, for example jexec and jspawnhelper are now stripped
> while the old strip logic missed them. I consider that fixing a bug.

Sure.

> I was about to leave this patch for another release but since the
> schedule was moved, and Magnus is available to review things again, I
> thought it was worth trying to get it in. I believe it will add
> significant value.

Ok.

>> An initial nit - why use "product-" in the target names? "images-jdk"
>> etc seems perfectly sufficient, and "product" may be a misnomer.
>>
> A while back when we started introducing other images (test-image,
> docs-image, symbols-image) we also renamed the old images target to
> product-images. We left the alias images though since it's fairly well
> established, and introduced the all-images target to build all of them.
> To me it was then natural to create sub targets of product-images rather
> than images for jdk and jre. I do think the new targets are too long
> however and because of that will likely not see much use. Perhaps we can
> add aliases for them too as you suggest.

Why not just jdk-image and jre-image?

Thanks,
David

> /Erik
>> Thanks,
>> David
>>
>> On 30/09/2016 8:17 PM, Erik Joelsson wrote:
>>> Here is a rather large patch which should make life better for most
>>> people building and developing OpenJDK. The main goal of this patch is
>>> to reduce unnecessary space used by the build. A side goal also become
>>> to figure out a better general strategy for handling native debug info
>>> during the build since that was a large part of the wasted space. These
>>> are the high level changes:
>>>
>>> 1. Removed the dist step in the hotspot build (and the corresponding
>>>    import step in the jdk build) and linking libjvm directly into
>>>    support/modules_libs. This removes 2 extra copies of all the hotspot
>>>    native libraries.
>>> 2. Made many files in the exploded image ($(OUTPUT_DIR)/jdk) symlinks
>>>    into support/modules_*. Note that binaries and libraries cannot be
>>>    symlinks since the image won't run then. Debug symbols and other
>>>    files seem to work fine though.
>>> 3. Introduced separate top level targets for building the jdk, jre (and
>>>    serverjre) images. A developer can run "make product-images-jdk" and
>>>    skip the others. The test targets now depend on just the jdk image
>>>    to avoid building extra images that aren't needed for tests.
>>> 4. Changed SetupNativeCompilation to strip binaries by default unless
>>>    told not to, and removed the separate strip build step. This lets us
>>>    get rid of modules_*-stripped.
>>> 5. Build debug symbols into the OUTPUT_DIR in SetupNativeCompilation
>>>    (instead of OBJECT_DIR and then copy to OUTPUT_DIR). For normal jdk
>>>    and hotspot libraries, this means the debug symbols end up directly
>>>    in support/modules_libs/$MODULE/... and there is only one copy of
>>> them.
>>> 6. If zipping debug symbols, this will also leave the unzipped
>>>    debugsymbols in OUTPUT_DIR, where we wouldn't do so before. This
>>>    also means that the exploded image will get symlinks to the unzipped
>>>    debug symbols regardless of if we are asked to zip them or not. This
>>>    in turn means that debugging the exploded image will just work. This
>>>    change is space neutral since we were already keeping a copy of the
>>>    unzipped debugsymbols in the OBJECT_DIR before, they have just been
>>>    moved to a more useful place.
>>> 7. Stop copying debug symbols from test native binaries since we aren't
>>>    stripping them anyway. Better to just leave debug information in the
>>>    test binaries.
>>>
>>> The reduction in size of course varies a lot depending on OS and
>>> configuration, but as an example, on my 64bit Linux workstation
>>> (debug-symbols=zipped) the size of the build directory has gone from
>>> 5.8GB when building "make images" before to 3.8GB when building "make
>>> product-images-jdk". The biggest gain is from only building the jdk
>>> image when only that is needed. A fairer comparison is "make images
>>> test-image" which has gone from 7.3GB to 5.4GB, much due to not
>>> duplicating debug information in test binaries. On Solaris the gain is
>>> generally bigger.
>>>
>>> Another nice effect of this is that the exploded image now has debug
>>> information for all native libraries by default, and that handling of
>>> dSYM directories on Macosx should now work properly.
>>>
>>> The only significant changes detected by comparison builds is that some
>>> binaries that previously weren't stripped, now are, which they also
>>> should be.
>>>
>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8150736
>>>
>>> Webrev: http://cr.openjdk.java.net/~erikj/8150736/webrev.01/
>>>
>>> /Erik
>>>
>