RFR: 8350148: Native stack overflow when writing Java heap objects into AOT cache

[Ioi Lam]

On Sun, 16 Feb 2025 05:16:17 GMT, Ioi Lam <iklam at openjdk.org> wrote:

> Please review this patch that fixes a problem that was found in the Leyden repo: when finding all cacheable heap objects with recursive calls to `HeapShared::archive_reachable_objects_from()`, very deep reference chains could cause overflow on the native stack. 
> 
> The fix is to do the recursion using a side table, without making any recursive calls.
> 
> Note: the kind of deep reference chains do not seem to happen with the mainline. It happened in the Leyden repo only after we enabled the caching of `WeakReference` objects, which are not cacheable in the mainline.

@shipilev @rose00 could you take a look?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/23654#issuecomment-2688987381