Problems with LTO for HotSpot

[Julian Waters]

Might there be any indication of what callees constitute the fast
paths, and which are slow paths (Or are in other compilation units)
and shouldn't be inlined? I've tried to determine that myself for
quite a while, but the call hierarchy is so unbelievably massive that
it's impossible to manually see which ones need to avoid inlining, and
all attempts to use tools to create a call graph have failed pretty
badly.

Maybe we could keep the flatten even with LTO if it's needed. I'd
hopefully like to avoid introducing an explicit "LTO enabled" marker
in autoconf and make beyond just setting the flags for LTO. But I
guess I'll cross that bridge when I come to it, right now I'm not even
able to get started trying to fix this issue.

best regards,
Julian

On Thu, Sep 25, 2025 at 6:16 AM Kim Barrett <kim.barrett at oracle.com> wrote:
>
> On 9/24/25 1:08 PM, Julian Waters wrote:
> > Recently I've been picking up and resuming work on making LTO viable
> > for HotSpot. [...] The bigger issue is related to the flatten attribute
> > on the gcc compiler. In short, some G1 code (More specifically void
> > G1ParScanThreadState::trim_queue_to_threshold(uint threshold), void
> > G1ParScanThreadState::steal_and_trim_queue(G1ScannerTasksQueueSet*
> > task_queues) and oop
> > G1ParScanThreadState::copy_to_survivor_space(G1HeapRegionAttr
> > region_attr, oop old, markWord old_mark)) is marked as flatten
>
> The flatten attribute is being used there because the code in question has
> been found to be very (performance) sensitive to compiler decisions to
> sometimes not inline something. So apply the big hammer.
>
> (Note that some critical path stuff is conditionally noinline in debug builds,
> because otherwise such builds were also running into excessive inlining,
> leading to things like "conditional branch out of range" failures.)
>
> Using flatten is certainly leading to more inlining than actually needed.
>
> An alternative to using flatten would be to try to mark all the critical path
> stuff always_inline. I found that pretty hard to do, and also brittle, much
> like Julian is encountering with the flatten + noinline approach.
>
> But it might be that some judicious always_inline + noinline + either (but not
> both) flatten or LTO might work. That was the direction I was going to
> explore, but haven't found any spare 'tuits to allocate in that direction.
> Supporting LTO just hasn't percolated up the priority list here.
>