Generational ZGC issue

[Johannes Lichtenberger]

Hello,

a test of my little DB project fails using generational ZGC, but not with
ZGC and G1 (out of memory error).

To be honest, I guess the allocation rate and thus GC pressure, when
traversing a resource in SirixDB is unacceptable. The strategy is to create
fine-grained nodes from JSON input and store these in a trie. First, a
3,8Gb JSON file is shredded and imported. Next, a preorder traversal of the
generated trie traverses a trie (with leaf pages storing 1024 nodes each
and in total ~300_000_000 (and these are going to be deserialized one by
one). The pages are furthermore referenced in memory through
PageReference::setPage. Furthermore, a Caffeine page cache caches the
PageReferences (keys) and the pages (values) and sets the reference back to
null once entries are going to be evicted (PageReference.setPage(null)).

However, I think the whole strategy of having to have in-memory nodes might
not be best. Maybe it's better to use off-heap memory for the pages itself
with MemorySegments, but the pages are not of a fixed size, thus it may get
tricky.

The test mentioned is this:
https://github.com/sirixdb/sirix/blob/248ab141632c94c6484a3069a056550516afb1d2/bundles/sirix-core/src/test/java/io/sirix/service/json/shredder/JsonShredderTest.java#L69

I can upload the JSON file somewhere for a couple of days if needed.

Caused by: java.lang.OutOfMemoryError
    at
java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
    at
java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
    at
java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
    at
java.base/java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:542)
    at
java.base/java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:567)
    at
java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:670)
    at
java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
    at
java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174)
    at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
    at
java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
    at
io.sirix.access.trx.page.NodePageTrx.parallelSerializationOfKeyValuePages(NodePageTrx.java:442)

I've uploaded several JFR recordings and logs over here (maybe besides the
async profiler JFR files the zgc-detailed log is most interesting):

https://github.com/sirixdb/sirix/tree/main/bundles/sirix-core

kind regards
Johannes
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/zgc-dev/attachments/20240214/bbc9f793/attachment-0001.htm>