RFR: 8342376: More reliable OOM handling in ExceptionDuringDumpAtObjectsInitPhase test

[Aleksey Shipilev]

On Wed, 16 Oct 2024 16:46:17 GMT, Ioi Lam <iklam at openjdk.org> wrote:

> Can you describe which test failed with Shenandoah and how the fail?

Looking deeper at this, I think the test is flaky through a different mechanism. The OOM that test wants to catch here:
https://github.com/openjdk/jdk/blob/d4f0ba73f653a3886b17f283b9b6a92db1af52aa/src/hotspot/share/cds/metaspaceShared.cpp#L664-L667

...comes from some other place, since we eat all exceptions from `ClassTransformer.transform` here:
https://github.com/openjdk/jdk/blob/d4f0ba73f653a3886b17f283b9b6a92db1af52aa/src/java.instrument/share/classes/sun/instrument/TransformerManager.java#L195-L198

I suspect there is some other allocation outside the transformer code gets the second "GC limit exceeded" OOM while allocating. Shenandoah does not handle GC limits, so it just happily allocates. Now, if we allocate 64K arrays in `Fill objects until OOM` block, OOM there, this means we still have 64KB of runway to work with. I believe this is enough for Shenandoah to complete the test without OOM. If I do just this:


-                    waste.add(new byte[64*1024]);
+                    waste.add(new byte[32*1024]);


Tests starts to pass with Shenandoah, which means 32KB free space was not enough to fly through the rest of the code.

I'll look again tomorrow.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/21536#issuecomment-2417666607