RFR: 8343698: Linux x86_64 lto build gives a lot of warnings and fails lto-wrapper: fatal error: make returned 2 exit status

[Matthias Baesken]

When trying LTO (configure flag --enable-jvm-feature-link-time-opt=yes) on Linux x86_64, gcc 11.3.0, we run into a lot of warnings and finally into this error :

 .. tons of free and malloc related warnings ...


    inlined from 'pop_segment' at src/hotspot/share/utilities/stack.inline.hpp:188:9,
    inlined from 'pop' at src/hotspot/share/utilities/stack.inline.hpp:84:30,
    inlined from 'pop_overflow' at src/hotspot/share/gc/shared/taskqueue.inline.hpp:231:28,
    inlined from 'trim_queue_to_threshold' at src/hotspot/share/gc/g1/g1ParScanThreadState.cpp:328:37:
src/hotspot/share/runtime/os.cpp:666:3: warning: call to 'malloc' declared with attribute warning: use os::malloc [-Wattribute-warning]
src/hotspot/share/runtime/os.cpp:666:3: warning: call to 'malloc' declared with attribute warning: use os::malloc [-Wattribute-warning]
In function 'malloc',
    inlined from 'malloc' at src/hotspot/share/runtime/os.cpp:637:0,
    inlined from 'AllocateHeap' at src/hotspot/share/memory/allocation.cpp:42:31,
    inlined from 'new_entry' at src/hotspot/share/nmt/mallocSiteTable.cpp:181:25,
    inlined from 'lookup_or_add' at src/hotspot/share/nmt/mallocSiteTable.cpp:122:48,
    inlined from 'allocation_at' at src/hotspot/share/nmt/mallocSiteTable.hpp:151:37,
    inlined from 'record_malloc' at src/hotspot/share/nmt/mallocTracker.cpp:179:35,
    inlined from 'record_malloc' at src/hotspot/share/nmt/memTracker.hpp:81:42,
    inlined from 'realloc' at src/hotspot/share/runtime/os.cpp:746:58,
    inlined from 'ReallocateHeap' at src/hotspot/share/memory/allocation.cpp:59:32,
    inlined from 'grow' at src/hotspot/share/utilities/ostream.cpp:377:15,
    inlined from 'write' at src/hotspot/share/utilities/ostream.cpp:400:11,
    inlined from 'write' at src/hotspot/share/utilities/ostream.cpp:382:0,
    inlined from 'put' at src/hotspot/share/utilities/ostream.cpp:212:8,
    inlined from 'print_ascii_form' at src/hotspot/share/runtime/os.cpp:965:19,
    inlined from 'print_hex_location' at src/hotspot/share/runtime/os.cpp:1008:21,
    inlined from 'print_hex_dump' at src/hotspot/share/runtime/os.cpp:1050:23,
    inlined from 'print_hex_dump' at src/hotspot/share/runtime/os.hpp:869:19,
    inlined from 'print_block_on_error' at src/hotspot/share/nmt/mallocHeader.cpp:64:23,
    inlined from 'resolve_checked_impl' at src/hotspot/share/nmt/mallocHeader.inline.hpp:106:41,
    inlined from 'resolve_checked' at src/hotspot/share/nmt/mallocHeader.inline.hpp:113:66,
    inlined from 'record_free_block' at src/hotspot/share/nmt/mallocTracker.cpp:206:55,
    inlined from 'record_free' at src/hotspot/share/nmt/memTracker.hpp:93:44,
    inlined from 'free' at src/hotspot/share/runtime/os.cpp:787:54,
    inlined from 'free' at src/hotspot/share/runtime/os.cpp:773:0,
    inlined from 'FreeHeap' at src/hotspot/share/memory/allocation.cpp:68:11,
    inlined from 'free' at src/hotspot/share/utilities/stack.inline.hpp:148:0,
    inlined from 'pop_segment' at src/hotspot/share/utilities/stack.inline.hpp:188:9,
    inlined from 'pop' at src/hotspot/share/utilities/stack.inline.hpp:84:30,
    inlined from 'pop_overflow' at src/hotspot/share/gc/shared/taskqueue.inline.hpp:231:28,
    inlined from 'trim_queue_to_threshold' at src/hotspot/share/gc/g1/g1ParScanThreadState.cpp:328:37:
src/hotspot/share/runtime/os.cpp:666:3: warning: call to 'malloc' declared with attribute warning: use os::malloc [-Wattribute-warning]
src/hotspot/share/runtime/os.cpp:666:3: warning: call to 'malloc' declared with attribute warning: use os::malloc [-Wattribute-warning]

src/hotspot/share/runtime/os.cpp:666:3: warning: call to 'malloc' declared with attribute warning: use os::malloc [-Wattribute-warning]
g++: fatal error: Killed signal terminated program lto1
compilation terminated.
make[4]: *** [/tmp/cc6nPei6.ltrans63.ltrans.o] Error 1
make[4]: *** Waiting for unfinished jobs....


Looks like the flattening attribute usage in src/hotspot/share/gc/g1/g1ParScanThreadState.cpp  causes the issues.
So it was discussed to remove the flattening attributes in this file but this was not seen as a good idea. So better disable  lto when building this special file with gcc .

-------------

Commit messages:
 - JDK-8343698

Changes: https://git.openjdk.org/jdk/pull/22069/files
  Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=22069&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8343698
  Stats: 6 lines in 2 files changed: 4 ins; 0 del; 2 mod
  Patch: https://git.openjdk.org/jdk/pull/22069.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/22069/head:pull/22069

PR: https://git.openjdk.org/jdk/pull/22069