RFR: 8370732: vmTestbase/nsk/jvmti/AttachOnDemand/attach020/TestDescription.java intermittent fails

[Chris Plummer]

On Tue, 28 Oct 2025 06:22:14 GMT, SendaoYan <syan at openjdk.org> wrote:

> Hi all,
> 
> We observed the test vmTestbase/nsk/jvmti/AttachOnDemand/attach020/TestDescription.java intermittent fails. since the target application intermittent OOM. The target application will try to exhaust the heap memory and then trigger garbage collection by `ClassUnloader.eatMemory()` in file test/hotspot/jtreg/vmTestbase/nsk/jvmti/AttachOnDemand/attach020/attach020Target.java.
> 
> So I think it's better to limited the max heap usage of target application. Before this PR, the physical memory usage is about 30.7GB(this depends the total free memory on the test machine), and the jtreg main action take about 11 seconds to finish; After this PR, the physical memory usage is about 126MB, and the jtreg main action take about 0.7 senonds to finish. I use below command to get the usage of physical memory.
> 
> Change has been verified locally, test-fix only, no risk.
> 
> 
> grep VmRSS /proc/`ps -ef | grep -w java | grep vmsqe.aod.AppId | awk '{print $2}'`/status

I don't see how your changes are fixing the problem described in the CR. The target process is exiting with an OOME, which shouldn't happen. The OOME is thrown and caught by ClassUnloader.eatMemory() and should not escape to the caller. I think your change is good practice for other reasons, but I don't see it resolving the failure, which I don't think is understood at this point.

A few other things of note about this test:

The test description does not mention provoking an OOME. It says it provokes a GC by calling System.gc(), so I'm not sure why it now calls ClassUnloader.eatMemory().

The test has exclusiveAccess.dirs=. in TEST.properties. This is probably because of the eatMemory() call and is no longer needed after your heap size change.

attach021 and attach022 look much like attach020 and could probably use the same improvements.

I see 6 other tests in nsk/jvmti that call eatMemory() and have TEST.properties. It seems these could all be improved.

-------------

PR Review: https://git.openjdk.org/jdk/pull/28011#pullrequestreview-3390601550