[jdk11u-dev] RFR: 8250597: G1: Improve inlining around trim_queue

[Dmitry Chuyko]

CC'ing hotspot-gc-dev.

On 6/11/21 5:08 PM, Dmitry Chuyko wrote:
> On Fri, 11 Jun 2021 13:53:36 GMT, Dmitry Chuyko <dchuyko at openjdk.org> wrote:
>
>> This change improves latency with G1 in some cases and makes trim_queue() code structure closer to current mainline.
>>
>> The original patch does not apply cleanly in particular because of https://bugs.openjdk.java.net/browse/JDK-8200545 . However the refactoring is rather simple because it is a rearrangement of code between functions with compiler hints. It has been reproduced (short description in the next comment).
>>
>> The change is isolated and is a set of code moves so the risk is low.
>>
>> Testing: tier1, tier2 on aarch64 and x86_64 in development, the change has been included in Liberica JDK 11u EA for some time.
>>
>> As an example of performance improvement, Critical-jOPS on Graviton2 raised +1%:+6.2% for 8GB heap.
> G1ParScanThreadState::trim_queue():
> Moved from g1ParScanThreadState.cpp to g1ParScanThreadState.inline.hpp, 'do-while !is_empty()' eleiminated.
>
> G1ParScanThreadState::do_oop_evac():
> Moved from g1ParScanThreadState.inline.hpp to g1ParScanThreadState.cpp, copy_to_survivor_space() -> do_copy_to_survivor_space()
>
> [G1ParScanThreadState::do_partial_array()]
> G1ParScanThreadState::do_oop_partial_array():
> Moved from g1ParScanThreadState.inline.hpp to g1ParScanThreadState.cpp, made not 'inline'.
>
> [G1ParScanThreadState::dispatch_task()]
> G1ParScanThreadState::dispatch_reference():
> Moved from g1ParScanThreadState.inline.hpp to g1ParScanThreadState.cpp, made not 'inline'.
>
> G1ParScanThreadState::needs_partial_trimming():
> Made not 'inline' in g1ParScanThreadState.inline.hpp.
>
> G1ParScanThreadState::trim_queue_to_threshold()
> Declaration moved from g1ParScanThreadState.inline.hpp to g1ParScanThreadState.cpp, 'inline' changed to 'ATTRIBUTE_FLATTEN NOINLINE' in g1ParScanThreadState.cpp, got 'do-while !overflow_empty()', made not 'inline' in g1ParScanThreadState.hpp.
>
> G1ParScanThreadState::steal_and_trim_queue():
> Declaration moved from g1ParScanThreadState.inline.hpp to g1ParScanThreadState.cpp,  with 'ATTRIBUTE_FLATTEN', comments, made not 'inline' in g1ParScanThreadState.hpp.
>
> G1ParScanThreadState::allocate_copy_slow():
> Added in g1ParScanThread_surviving_young_words[young_index] += word_szState.cpp as 'NOINLINE', extracted from copy_to_survivor_space/if (obj_ptr == NULL) except the case when handle_evacuation_failure_par() is needed, part before tracer is wrapped in if (!(dest_attr->is_old() && _old_gen_is_full)), tracing is wrapped in if (obj_ptr != NULL), declaration added in g1ParScanThreadState.hpp.
>
> G1ParScanThreadState::undo_allocation():
> Added in g1ParScanThreadState.cpp as 'NOINLINE', extracted from copy_to_survivor_space/_plab_allocator->undo_allocation, declaration added in g1ParScanThreadState.hpp.
>
> G1ParScanThreadState::copy_to_survivor_space():
> Renamed to do_copy_to_survivor_space() in g1ParScanThreadState.cpp + comment, copy_to_survivor_space() added as a wrapper in g1ParScanThreadState.cpp, do_copy_to_survivor_space() declaration added and copy_to_survivor_space() declaration corrected in g1ParScanThreadState.hpp.
> assert(is_in_cset()) has been added.
> if(_old_gen_is_full && dest_state.is_old()) block moved.
> _surviving_young_words[young_index] += word_sz moved into new block around young_index >  0 asserts
>
> G1ParScanThreadState::handle_evacuation_failure_par():
> 'NOINLINE' in g1ParScanThreadState.cpp.
>
> G1ParScanThreadState::is_partially_trimmed():
> Manually inlined into trim_queue_partially().
>
> G1ParScanThreadState::trim_queue_partially():
> 'inline' in g1ParScanThreadState.hpp instead of g1ParScanThreadState.inline.hpp.
> do-while moved (in the patch it is replaced by asserts).
>
> G1ParScanThreadState::trim_queue():
> Made 'inline' in g1ParScanThreadState.hpp.
>
> ATTRIBUTE_FLATTEN
> Defined in globalDefinitions.hpp, globalDefinitions_gcc.hpp.
>
> Manually inline deal_with_reference()-s into G1ParScanThreadState::dispatch_reference().
>
> -------------
>
> PR: https://git.openjdk.java.net/jdk11u-dev/pull/26