RFR: 8307958: Metaspace verification is slow causing extreme class unloading times

[Thomas Stuefe]

On Mon, 22 May 2023 14:22:07 GMT, Axel Boldt-Christmas <aboldtch at openjdk.org> wrote:

> From JBS:
>>We see very long class unloading times with Generational ZGC in fastdebug builds. MacOS spindumps show that it is caused by Metaspace verification code.
>>
>>I think the main problem is that many of the verification operations are performed at every operation (add, remove, split, merge) and the individual operation verifies much more than a constant number of chunks. This seems to give a quadratic complexity which blows up if you don't class unload often enough.
> 
> The approach here is to resolve this by putting the verification behind the SOMETIMES macro. It is then possible to turn it of completely with `-XX:VerifyMetaspaceInterval=0` while still benefiting from some stochastic verification. 
> 
> The goal here is to have the calls to Verify unchanged. Leaving all explicit gtest verify and Universe::verify calls unaffected.  One gtest(`metaspace.test_overwriter_vm_assert`) had an implicit dependency on the `~MetaspaceArena` verification, so it was moved to `~MetaspaceTestArena` which now performs the verification explicitly.  It becomes somewhat weaker, it does not test that the destructor SOMETIMES verifies, but that if it verifies it would get caught. Hopefully this is alright.
> The verification was also left unchanged when committing the memory. 
> 
> All SOMETIMES uses the same frequency, `VerifyMetaspaceInterval`. I looked at the possibility of adding weight to some of the call site, but it did not seem to be worth the complexity. 
> 
> Testing: GHA and Oracle CI (tier1-3 ongoing), various long running stress tests.

Seems reasonable.

Could you please crank up the verification interval for metaspace gtests from 3 to 1? (See test/hotspot/jtreg/gtest/MetaspaceGtests.java).

-------------

Changes requested by stuefe (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/14084#pullrequestreview-1436805162