RFR: 8375359: Improve GC serviceability init staging

[Stefan Karlsson]

On Thu, 15 Jan 2026 16:43:40 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> As per docs, `CollectedHeap::post_initialize` is for doing inits that should be done before general allocations are accepted. Currently, serviceability layer initialized there via the call to `CollectedHeap::initialize_serviceability`. This bug shows that after `CollectedHeap::post_initialize` returns, the rest of the serviceability layer had not been fully initialized yet. But GC code might assume general allocation path works at that point! So if that allocation path reports anything to the serviceability layer, it could encounter not yet fully initialized serviceability support and fail.
> 
> This readily manifests in improved Epsilon stress test. Improved test fails pretty consistently on my machines, and serves as a good regression test.
> 
> So the fix is to peel off serviceability initialization from post-init stage, and do it separately, before going (and finishing) `post_initialize`. Conveniently, we are nearly there, and "just" need to touch up a few things in `universe_post_init()`.
> 
> Additional testing:
>  - [x] Linux x86_64 server fastdebug, `gc/epsilon/TestInitAllocs.java`, 10x
>  - [x] Linux x86_64 server fastdebug, `hotspot_gc`

Is there a way to restructure the code so that we don't get more locations in the VM initialization code (external to the GC code) that call various CollectedHeap initialization functions?

I haven't fully looked at this in all details, but could the various `initialize_serviceability` function be called from within the `<GC>CollectedHeap::initialize` function instead?

-------------

PR Review: https://git.openjdk.org/jdk/pull/29254#pullrequestreview-3705800541