<div dir="auto">Thanks a lot, I wasn't even aware of the fact, that DirectByteBuffers use System.gc() and I always had in mind that calling System.gc() at least in application code is bad practice (or at least we shouldn't rely on it) and I think I read somewhere a while ago, that it's recommended to even disable this, but may be completely wrong, of course.<div dir="auto"><br></div><div dir="auto">I'll change it to on-heap byte buffers tomorrow :-)</div><div dir="auto"><br></div><div dir="auto">I think your GC log entries were from G1, right? It seems ZGC always tries to use the full heap :-)</div><div dir="auto"><br></div><div dir="auto">Kind regards and thanks for sharing your insights.</div><div dir="auto"><br></div><div dir="auto">Have a nice weekend as well</div><div dir="auto">Johannes</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Stefan Johansson <<a href="mailto:stefan.johansson@oracle.com">stefan.johansson@oracle.com</a>> schrieb am Fr., 16. Feb. 2024, 17:38:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
Some comments inline.<br>
<br>
On 2024-02-16 16:47, Johannes Lichtenberger wrote:<br>
> Thanks a lot for looking into it, I've added <br>
> `-XX:MaxDirectMemorySize=2g` only recently, but without it failed as <br>
> well, so not sure what the default is. Will definitely check your <br>
> suggestions :-)<br>
> <br>
If you don't set a limit it will be set to:<br>
Runtime.getRuntime().maxMemory()<br>
So likely a good idea to set a reasonable limit, but the smaller the <br>
limit is the more frequent we need to run reference processing to allow <br>
memory to be freed up.<br>
<br>
> Sadly I'm currently working alone on the project in my spare time <br>
> (besides professionally switched from Java/Kotlin stuff to the embedded <br>
> software world) and I'm not sure if the current architecture of Sirix is <br>
> limited by too much GC pressure. I'd probably have to check Cassandra at <br>
> some point and look into flame graphs and stuff for their integration <br>
> tests, but maybe you can give some general insights/advice...<br>
> <br>
> Yesterday evening I switched to other JDKs (also I want to test with <br>
> Shenandoah in particular), but I think especially the better escape <br>
> analysis of the GraalVM is a huge plus in the case of SirixDB (for <br>
> insertion on my laptop it's ~90s vs ~60s), but I think it should be <br>
> faster and currently my suspicion is that garbage is a major performance <br>
> issue.<br>
> <br>
> Maybe the GC pressure in general is a major issue, as in the CPU Flame <br>
> graph IIRC the G1 had about 20% stack frames allocated and non <br>
> generational ZGC even around 40% taking all threads into account.<br>
> <br>
<br>
From what I/we see, the GC pressure in the given test is not high. The <br>
allocation rate is below 1GB/s and since most of it die young the GCs <br>
are fairly cheap. In this log snippet G1 shows a GC every 5s and the <br>
pause time is below 50ms:<br>
[296,016s][info ][gc ] GC(90) Pause Young (Normal) (G1 Evacuation <br>
Pause) 5413M->1849M(6456M) 35,577ms<br>
[301,103s][info ][gc ] GC(91) Pause Young (Normal) (G1 Evacuation <br>
Pause) 5417M->1848M(6456M) 33,357ms<br>
[306,041s][info ][gc ] GC(92) Pause Young (Normal) (G1 Evacuation <br>
Pause) 5416M->1848M(6456M) 32,763ms<br>
[310,849s][info ][gc ] GC(93) Pause Young (Normal) (G1 Evacuation <br>
Pause) 5416M->1847M(6456M) 33,086ms<br>
<br>
I also see that the heap never expands to more the ~6.5GB even though it <br>
is allow to be 15GB and this also suggest that the GC is not under much <br>
pressure. As I said in the previous mail, the reason Generational ZGC <br>
don't free up the direct memory without the System.gc() calls is that <br>
the GC pressure is not high enough to trigger any Major cycles. So I <br>
would strongly recommend you to not run with -XX+DisableExplicitGC <br>
unless you really have to. Since you are using DirectByteBuffers and <br>
they use System.gc() to help free memory when the limit is reached.<br>
<br>
> So in general I'm thinking about backing the KeyValueLeafPages with <br>
> MemorySegments, but I think due to variable sized pages it's getting <br>
> tricky, plus I currently don't have the time for changing fundamental <br>
> stuff and I'm even not sure if it'll bring a performance boost, as I <br>
> have to adapt neighbour relationships of the nodes often and off-heap <br>
> memory access might be slightly worse performance wise.<br>
> <br>
> What do you think?<br>
> <br>
<br>
I know to little about the application to be able to give advice here, <br>
but I would first start with having most memory on heap. Only large long <br>
lived stuff off-heap, if really needed. Looking at the test at hand, it <br>
really doesn't look like it is long lived stuff that is placed off heap.<br>
<br>
> I've attached a memory flame graph and there it seems the byte array <br>
> from deserializing each page is prominent, but that might be something I <br>
> can't even avoid (after decompression via Snappy or via another lib and <br>
> maybe also decryption in the future).<br>
> <br>
> As of now G1 with GraalVM seems to perform best (a little bit better <br>
> than with non generational ZGC, but I thought ZGC or maybe Shenandoah <br>
> would improve the situation). But as said I may have to generate way <br>
> less garbage after all in general for good performance!?<br>
> <br>
> All in all maybe due to most objects die young maybe also the <br>
> generational GCs are not needed (that said if enough RAM is available <br>
> and the Caffeine Caches are sized accordingly most objects may die old). <br>
> But apparently the byte arrays holding the page data still die young (in <br>
> AbstractReader::deserialize). In fact I'm not even sure why they escape, <br>
> but currently I'm on my phone.<br>
> <br>
<br>
It's when most objects die young the Generational GC really shines, <br>
because it can handle the short lived objects without having to look at <br>
the long lived objects. So I would say Generational ZGC is a good fit <br>
here, but we need to let the System.gc() run to allow reference <br>
processing or slightly re-design and use HeapByteBuffers.<br>
<br>
Have a nice weekend,<br>
Stefan<br>
<br>
> Kind regards<br>
> Johannes<br>
> <br>
> Stefan Johansson <<a href="mailto:stefan.johansson@oracle.com" target="_blank" rel="noreferrer">stefan.johansson@oracle.com</a> <br>
> <mailto:<a href="mailto:stefan.johansson@oracle.com" target="_blank" rel="noreferrer">stefan.johansson@oracle.com</a>>> schrieb am Fr., 16. Feb. 2024, 13:43:<br>
> <br>
> Hi Johannes,<br>
> <br>
> We've spent some more time looking at this and getting the json-file to<br>
> reproduced it made it easy to verify our suspicions. Thanks for<br>
> uploading it.<br>
> <br>
> There are a few things playing together here. The test is making quite<br>
> heavy use of DirectByteBuffers and you limit the usage to 2G<br>
> (-XX:MaxDirectMemorySize=2g). The life cycle and freeing of the native<br>
> memory part of the DirectByteBuffer rely on reference processing. In<br>
> generational ZGC reference processing is only done during Major<br>
> collections and since the general GC preassure in this benchmark is<br>
> very<br>
> low (most objects die young), we do not trigger that many Major<br>
> collections.<br>
> <br>
> Normaly this would not be a problem. To avoid throwing an out of memory<br>
> error (due to hitting the direct buffer memory limit) too early the JDK<br>
> triggers a System.gc(). This should trigger reference procesing and all<br>
> buffers that are no longer in use would be freed. Since you specify the<br>
> option -XX:+DisableExplicitGC all these calls to trigger GCs are<br>
> ignored<br>
> and no direct memory will be freed. So in our testing, just removing<br>
> this flags makes the test pass.<br>
> <br>
> Another solution is to look at using HeapByteBuffers instead and don't<br>
> have to worry about the direct memory usage. The OpenHFT lib seems to<br>
> have support for this by just using elasticHeapByteBuffer(...) instead<br>
> of elasticByteBuffer().<br>
> <br>
> Lastly, the reason for this working with non-generational ZGC is<br>
> that it<br>
> does reference processing for every GC.<br>
> <br>
> Hope this helps,<br>
> StefanJ<br>
> <br>
> <br>
> On 2024-02-15 21:53, Johannes Lichtenberger wrote:<br>
> > It's a laptop, I've attached some details.<br>
> ><br>
> > Furthermore, if it seems worth digging deeper into the issue, the<br>
> JSON<br>
> > file is here for one week:<br>
> > <a href="https://www.transfernow.net/dl/20240215j9NaPTc0" rel="noreferrer noreferrer" target="_blank">https://www.transfernow.net/dl/20240215j9NaPTc0</a><br>
> <<a href="https://urldefense.com/v3/__https://www.transfernow.net/dl/20240215j9NaPTc0__;!!ACWV5N9M2RV99hQ!MWZDuvCBsbZSYul-HLDtF_j1IBD6osBF4cBVE_bg0yM5zCqYFwzLLp7nKN3b1hq1XVFRreqUVaXiKuXjUwGbxpjjFeDXD5_-$" rel="noreferrer noreferrer" target="_blank">https://urldefense.com/v3/__https://www.transfernow.net/dl/20240215j9NaPTc0__;!!ACWV5N9M2RV99hQ!MWZDuvCBsbZSYul-HLDtF_j1IBD6osBF4cBVE_bg0yM5zCqYFwzLLp7nKN3b1hq1XVFRreqUVaXiKuXjUwGbxpjjFeDXD5_-$</a>><br>
> > <<a href="https://www.transfernow.net/dl/20240215j9NaPTc0" rel="noreferrer noreferrer" target="_blank">https://www.transfernow.net/dl/20240215j9NaPTc0</a><br>
> <<a href="https://urldefense.com/v3/__https://www.transfernow.net/dl/20240215j9NaPTc0__;!!ACWV5N9M2RV99hQ!MWZDuvCBsbZSYul-HLDtF_j1IBD6osBF4cBVE_bg0yM5zCqYFwzLLp7nKN3b1hq1XVFRreqUVaXiKuXjUwGbxpjjFeDXD5_-$" rel="noreferrer noreferrer" target="_blank">https://urldefense.com/v3/__https://www.transfernow.net/dl/20240215j9NaPTc0__;!!ACWV5N9M2RV99hQ!MWZDuvCBsbZSYul-HLDtF_j1IBD6osBF4cBVE_bg0yM5zCqYFwzLLp7nKN3b1hq1XVFRreqUVaXiKuXjUwGbxpjjFeDXD5_-$</a>>><br>
> ><br>
> > You'd have to unzip into bundles/sirix-core/src/test/resources/json,<br>
> > remove the @Disabled annotation and run the test<br>
> > JsonShredderTest::testChicagoDescendantAxis<br>
> ><br>
> > The test JVM parameters are specified in the parent build.gradle<br>
> in the<br>
> > project root folder.<br>
> ><br>
> > The GitHub repo: <a href="https://github.com/sirixdb/sirix" rel="noreferrer noreferrer" target="_blank">https://github.com/sirixdb/sirix</a><br>
> <<a href="https://urldefense.com/v3/__https://github.com/sirixdb/sirix__;!!ACWV5N9M2RV99hQ!MWZDuvCBsbZSYul-HLDtF_j1IBD6osBF4cBVE_bg0yM5zCqYFwzLLp7nKN3b1hq1XVFRreqUVaXiKuXjUwGbxpjjFUUPdeUD$" rel="noreferrer noreferrer" target="_blank">https://urldefense.com/v3/__https://github.com/sirixdb/sirix__;!!ACWV5N9M2RV99hQ!MWZDuvCBsbZSYul-HLDtF_j1IBD6osBF4cBVE_bg0yM5zCqYFwzLLp7nKN3b1hq1XVFRreqUVaXiKuXjUwGbxpjjFUUPdeUD$</a>><br>
> > <<a href="https://github.com/sirixdb/sirix" rel="noreferrer noreferrer" target="_blank">https://github.com/sirixdb/sirix</a><br>
> <<a href="https://urldefense.com/v3/__https://github.com/sirixdb/sirix__;!!ACWV5N9M2RV99hQ!MWZDuvCBsbZSYul-HLDtF_j1IBD6osBF4cBVE_bg0yM5zCqYFwzLLp7nKN3b1hq1XVFRreqUVaXiKuXjUwGbxpjjFUUPdeUD$" rel="noreferrer noreferrer" target="_blank">https://urldefense.com/v3/__https://github.com/sirixdb/sirix__;!!ACWV5N9M2RV99hQ!MWZDuvCBsbZSYul-HLDtF_j1IBD6osBF4cBVE_bg0yM5zCqYFwzLLp7nKN3b1hq1XVFRreqUVaXiKuXjUwGbxpjjFUUPdeUD$</a>>><br>
> ><br>
> > Screenshot from 2024-02-15 21-43-33.png<br>
> ><br>
> > kind regards<br>
> > Johannes<br>
> ><br>
> > Am Do., 15. Feb. 2024 um 20:01 Uhr schrieb Peter Booth<br>
> > <<a href="mailto:peter_booth@me.com" target="_blank" rel="noreferrer">peter_booth@me.com</a> <mailto:<a href="mailto:peter_booth@me.com" target="_blank" rel="noreferrer">peter_booth@me.com</a>><br>
> <mailto:<a href="mailto:peter_booth@me.com" target="_blank" rel="noreferrer">peter_booth@me.com</a> <mailto:<a href="mailto:peter_booth@me.com" target="_blank" rel="noreferrer">peter_booth@me.com</a>>>>:<br>
> ><br>
> > Just curious - what CPU, physical memory and OS are you using?<br>
> > Sent from my iPhone<br>
> ><br>
> >> On Feb 15, 2024, at 12:23 PM, Johannes Lichtenberger<br>
> >> <<a href="mailto:lichtenberger.johannes@gmail.com" target="_blank" rel="noreferrer">lichtenberger.johannes@gmail.com</a><br>
> <mailto:<a href="mailto:lichtenberger.johannes@gmail.com" target="_blank" rel="noreferrer">lichtenberger.johannes@gmail.com</a>><br>
> >> <mailto:<a href="mailto:lichtenberger.johannes@gmail.com" target="_blank" rel="noreferrer">lichtenberger.johannes@gmail.com</a><br>
> <mailto:<a href="mailto:lichtenberger.johannes@gmail.com" target="_blank" rel="noreferrer">lichtenberger.johannes@gmail.com</a>>>> wrote:<br>
> >><br>
> >> <br>
> >> I guess I don't know which JDK it picks for the tests, but I<br>
> guess<br>
> >> OpenJDK<br>
> >><br>
> >> Johannes Lichtenberger <<a href="mailto:lichtenberger.johannes@gmail.com" target="_blank" rel="noreferrer">lichtenberger.johannes@gmail.com</a><br>
> <mailto:<a href="mailto:lichtenberger.johannes@gmail.com" target="_blank" rel="noreferrer">lichtenberger.johannes@gmail.com</a>><br>
> >> <mailto:<a href="mailto:lichtenberger.johannes@gmail.com" target="_blank" rel="noreferrer">lichtenberger.johannes@gmail.com</a><br>
> <mailto:<a href="mailto:lichtenberger.johannes@gmail.com" target="_blank" rel="noreferrer">lichtenberger.johannes@gmail.com</a>>>> schrieb am Do., 15.<br>
> >> Feb. 2024, 17:58:<br>
> >><br>
> >> However, it's the same with: ./gradlew<br>
> >> -Dorg.gradle.java.home=/home/johannes/.jdks/openjdk-21.0.2<br>
> >> :sirix-core:test --tests<br>
> >> <br>
> io.sirix.service.json.shredder.JsonShredderTest.testChicagoDescendantAxis using OpenJDK hopefully<br>
> >><br>
> >> Am Do., 15. Feb. 2024 um 17:54 Uhr schrieb Johannes<br>
> >> Lichtenberger <<a href="mailto:lichtenberger.johannes@gmail.com" target="_blank" rel="noreferrer">lichtenberger.johannes@gmail.com</a><br>
> <mailto:<a href="mailto:lichtenberger.johannes@gmail.com" target="_blank" rel="noreferrer">lichtenberger.johannes@gmail.com</a>><br>
> >> <mailto:<a href="mailto:lichtenberger.johannes@gmail.com" target="_blank" rel="noreferrer">lichtenberger.johannes@gmail.com</a><br>
> <mailto:<a href="mailto:lichtenberger.johannes@gmail.com" target="_blank" rel="noreferrer">lichtenberger.johannes@gmail.com</a>>>>:<br>
> >><br>
> >> I've attached two logs, the first one without<br>
> >> -XX:+Generational, the second one with the option set,<br>
> >> even though I also saw, that generational ZGC is<br>
> going to<br>
> >> be supported in GraalVM 24.1 in September... so not sure<br>
> >> what this does :)<br>
> >><br>
> >> Am Do., 15. Feb. 2024 um 17:52 Uhr schrieb Johannes<br>
> >> Lichtenberger <<a href="mailto:lichtenberger.johannes@gmail.com" target="_blank" rel="noreferrer">lichtenberger.johannes@gmail.com</a><br>
> <mailto:<a href="mailto:lichtenberger.johannes@gmail.com" target="_blank" rel="noreferrer">lichtenberger.johannes@gmail.com</a>><br>
> >> <mailto:<a href="mailto:lichtenberger.johannes@gmail.com" target="_blank" rel="noreferrer">lichtenberger.johannes@gmail.com</a><br>
> <mailto:<a href="mailto:lichtenberger.johannes@gmail.com" target="_blank" rel="noreferrer">lichtenberger.johannes@gmail.com</a>>>>:<br>
> >><br>
> >> Strange, so does it simply ignore the option? The<br>
> >> following is the beginning of the output from _non_<br>
> >> generational ZGC:<br>
> >><br>
> >> johannes@luna:~/IdeaProjects/sirix$ ./gradlew<br>
> >> <br>
> -Dorg.gradle.java.home=/home/johannes/.sdkman/candidates/java/21.0.2-graal :sirix-core:test --tests io.sirix.service.json.shredder.JsonShredderTest.testChicagoDescendantAxis<br>
> >><br>
> >> > Configure project :<br>
> >> The 'sonarqube' task depends on compile tasks. This<br>
> >> behavior is now deprecated and will be removed in<br>
> >> version 5.x. To avoid implicit compilation, set<br>
> >> property 'sonar.gradle.skipCompile' to 'true'<br>
> and make<br>
> >> sure your project is compiled, before analysis has<br>
> >> started.<br>
> >> The 'sonar' task depends on compile tasks. This<br>
> >> behavior is now deprecated and will be removed in<br>
> >> version 5.x. To avoid implicit compilation, set<br>
> >> property 'sonar.gradle.skipCompile' to 'true'<br>
> and make<br>
> >> sure your project is compiled, before analysis has<br>
> >> started.<br>
> >> [1,627s][info ][gc ] GC(0) Garbage Collection<br>
> >> (Metadata GC Threshold) 84M(1%)->56M(0%)<br>
> >><br>
> >> > Task :sirix-core:test<br>
> >> [0.001s][warning][pagesize] UseLargePages<br>
> disabled, no<br>
> >> large pages configured and available on the system.<br>
> >> [1.253s][info ][gc ] Using The Z Garbage<br>
> Collector<br>
> >><br>
> >> [2,930s][info ][gc ] GC(1) Garbage Collection<br>
> >> (Warmup) 1616M(11%)->746M(5%)<br>
> >> [4,445s][info ][gc ] GC(2) Garbage Collection<br>
> >> (Warmup) 3232M(21%)->750M(5%)<br>
> >> [5,751s][info ][gc ] GC(3) Garbage Collection<br>
> >> (Warmup) 4644M(30%)->1356M(9%)<br>
> >> [9,886s][info ][gc ] GC(4) Garbage Collection<br>
> >> (Allocation Rate) 10668M(69%)->612M(4%)<br>
> >> [10,406s][info ][gc ] GC(5) Garbage<br>
> Collection<br>
> >> (Allocation Rate) 2648M(17%)->216M(1%)<br>
> >> [13,931s][info ][gc ] GC(6) Garbage<br>
> Collection<br>
> >> (Allocation Rate) 11164M(73%)->1562M(10%)<br>
> >> [16,908s][info ][gc ] GC(7) Garbage<br>
> Collection<br>
> >> (Allocation Rate) 11750M(76%)->460M(3%)<br>
> >> [20,690s][info ][gc ] GC(8) Garbage<br>
> Collection<br>
> >> (Allocation Rate) 12670M(82%)->726M(5%)<br>
> >> [24,376s][info ][gc ] GC(9) Garbage<br>
> Collection<br>
> >> (Allocation Rate) 13422M(87%)->224M(1%)<br>
> >> [28,152s][info ][gc ] GC(10) Garbage<br>
> Collection<br>
> >> (Proactive) 13474M(88%)->650M(4%)<br>
> >> [31,526s][info ][gc ] GC(11) Garbage<br>
> Collection<br>
> >> (Allocation Rate) 12072M(79%)->1472M(10%)<br>
> >> [34,754s][info ][gc ] GC(12) Garbage<br>
> Collection<br>
> >> (Allocation Rate) 13050M(85%)->330M(2%)<br>
> >> [38,478s][info ][gc ] GC(13) Garbage<br>
> Collection<br>
> >> (Allocation Rate) 13288M(87%)->762M(5%)<br>
> >> [41,936s][info ][gc ] GC(14) Garbage<br>
> Collection<br>
> >> (Proactive) 13294M(87%)->504M(3%)<br>
> >> [45,353s][info ][gc ] GC(15) Garbage<br>
> Collection<br>
> >> (Allocation Rate) 12984M(85%)->268M(2%)<br>
> >> [48,861s][info ][gc ] GC(16) Garbage<br>
> Collection<br>
> >> (Allocation Rate) 13008M(85%)->306M(2%)<br>
> >> [52,133s][info ][gc ] GC(17) Garbage<br>
> Collection<br>
> >> (Proactive) 12042M(78%)->538M(4%)<br>
> >> [55,705s][info ][gc ] GC(18) Garbage<br>
> Collection<br>
> >> (Allocation Rate) 12420M(81%)->1842M(12%)<br>
> >> [59,000s][info ][gc ] GC(19) Garbage<br>
> Collection<br>
> >> (Allocation Rate) 12458M(81%)->1422M(9%)<br>
> >> [64,501s][info ][gc ] Allocation Stall (Test<br>
> >> worker) 59,673ms<br>
> >> [64,742s][info ][gc ] Allocation Stall (Test<br>
> >> worker) 240,077ms<br>
> >> [65,806s][info ][gc ] GC(20) Garbage<br>
> Collection<br>
> >> (Allocation Rate) 13808M(90%)->6936M(45%)<br>
> >> [66,476s][info ][gc ] GC(21) Garbage<br>
> Collection<br>
> >> (Allocation Stall) 7100M(46%)->4478M(29%)<br>
> >> [69,471s][info ][gc ] GC(22) Garbage<br>
> Collection<br>
> >> (Allocation Rate) 10098M(66%)->5888M(38%)<br>
> >> [72,252s][info ][gc ] GC(23) Garbage<br>
> Collection<br>
> >> (Allocation Rate) 11226M(73%)->5816M(38%)<br>
> >><br>
> >> ...<br>
> >><br>
> >> So even here I can see some allocation stalls.<br>
> >><br>
> >> Running the Same with -XX:+ZGenerational in<br>
> >> build.gradle probably using GraalVM does something<br>
> >> differnt, but I don't know what... at least off-heap<br>
> >> memory is exhausted at some point due to direct byte<br>
> >> buffer usage!?<br>
> >><br>
> >> So, I'm not sure what's the difference, though.<br>
> >><br>
> >> With this:<br>
> >><br>
> >> "-XX:+UseZGC",<br>
> >> <br>
> "-Xlog:gc*=debug:file=zgc-generational-detailed.log",<br>
> >> "-XX:+ZGenerational",<br>
> >> "-verbose:gc",<br>
> >> "-XX:+HeapDumpOnOutOfMemoryError",<br>
> >> "-XX:HeapDumpPath=heapdump.hprof",<br>
> >> "-XX:MaxDirectMemorySize=2g",<br>
> >><br>
> >><br>
> >> Caused by: java.lang.OutOfMemoryError: Cannot<br>
> reserve 60000 bytes of direct buffer memory (allocated: 2147446560,<br>
> limit: 2147483648)<br>
> >> at<br>
> java.base/java.nio.Bits.reserveMemory(Bits.java:178)<br>
> >> at<br>
> java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:111)<br>
> >> at<br>
> java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:360)<br>
> >> at<br>
> net.openhft.chronicle.bytes.internal.NativeBytesStore.elasticByteBuffer(NativeBytesStore.java:191)<br>
> >> at<br>
> net.openhft.chronicle.bytes.BytesStore.elasticByteBuffer(BytesStore.java:192)<br>
> >> at<br>
> net.openhft.chronicle.bytes.Bytes.elasticByteBuffer(Bytes.java:176)<br>
> >> at<br>
> net.openhft.chronicle.bytes.Bytes.elasticByteBuffer(Bytes.java:148)<br>
> >> at <a href="http://io.sirix.access.trx.page" rel="noreferrer noreferrer" target="_blank">io.sirix.access.trx.page</a><br>
> <<a href="https://urldefense.com/v3/__http://io.sirix.access.trx.page__;!!ACWV5N9M2RV99hQ!MWZDuvCBsbZSYul-HLDtF_j1IBD6osBF4cBVE_bg0yM5zCqYFwzLLp7nKN3b1hq1XVFRreqUVaXiKuXjUwGbxpjjFeoArpQf$" rel="noreferrer noreferrer" target="_blank">https://urldefense.com/v3/__http://io.sirix.access.trx.page__;!!ACWV5N9M2RV99hQ!MWZDuvCBsbZSYul-HLDtF_j1IBD6osBF4cBVE_bg0yM5zCqYFwzLLp7nKN3b1hq1XVFRreqUVaXiKuXjUwGbxpjjFeoArpQf$</a>>.NodePageTrx.lambda$parallelSerializationOfKeyValuePages$1(NodePageTrx.java:443)<br>
> >><br>
> >><br>
> >><br>
> >> Am Do., 15. Feb. 2024 um 12:05 Uhr schrieb Stefan<br>
> >> Karlsson <<a href="mailto:stefan.karlsson@oracle.com" target="_blank" rel="noreferrer">stefan.karlsson@oracle.com</a><br>
> <mailto:<a href="mailto:stefan.karlsson@oracle.com" target="_blank" rel="noreferrer">stefan.karlsson@oracle.com</a>><br>
> >> <mailto:<a href="mailto:stefan.karlsson@oracle.com" target="_blank" rel="noreferrer">stefan.karlsson@oracle.com</a><br>
> <mailto:<a href="mailto:stefan.karlsson@oracle.com" target="_blank" rel="noreferrer">stefan.karlsson@oracle.com</a>>>>:<br>
> >><br>
> >> Hi Johannes,<br>
> >><br>
> >> We tried to look at the log files and the jfr<br>
> >> files, but couldn't find<br>
> >> an OotOfMemoryError in any of them. Do you think<br>
> >> you could try to rerun<br>
> >> and capture the entire GC log from the<br>
> >> OutOfMemoryError run?<br>
> >><br>
> >> A few things to note:<br>
> >><br>
> >> 1) You seem to be running the Graal compiler.<br>
> >> Graal doesn't support<br>
> >> Generational ZGC, so you are going to run<br>
> >> different compilers when you<br>
> >> compare Singlegen ZGC with Generational ZGC.<br>
> >><br>
> >> 2) It's not clear to me that the provided JFR<br>
> >> files matches the provided<br>
> >> log files.<br>
> >><br>
> >> 3) The JFR files show that<br>
> -XX:+UseLargePages are<br>
> >> used, but the gc+init<br>
> >> logs shows 'Large Page Support: Disabled', you<br>
> >> might want to look into<br>
> >> why that is the case.<br>
> >><br>
> >> 4) The singlegen JFR file has a<br>
> >> -Xlog:gc:g1-chicago.log line. It should<br>
> >> probably be named zgc-chicago.log.<br>
> >><br>
> >> Cheers,<br>
> >> StefanK<br>
> >><br>
> >> On 2024-02-14 17:36, Johannes Lichtenberger<br>
> wrote:<br>
> >> > Hello,<br>
> >> ><br>
> >> > a test of my little DB project fails using<br>
> >> generational ZGC, but not<br>
> >> > with ZGC and G1 (out of memory error).<br>
> >> ><br>
> >> > To be honest, I guess the allocation rate and<br>
> >> thus GC pressure, when<br>
> >> > traversing a resource in SirixDB is<br>
> >> unacceptable. The strategy is to<br>
> >> > create fine-grained nodes from JSON input and<br>
> >> store these in a trie.<br>
> >> > First, a 3,8Gb JSON file is shredded and<br>
> >> imported. Next, a preorder<br>
> >> > traversal of the generated trie traverses<br>
> a trie<br>
> >> (with leaf pages<br>
> >> > storing 1024 nodes each and in total<br>
> >> ~300_000_000 (and these are going<br>
> >> > to be deserialized one by one). The pages are<br>
> >> furthermore referenced<br>
> >> > in memory through PageReference::setPage.<br>
> >> Furthermore, a Caffeine page<br>
> >> > cache caches the PageReferences (keys) and the<br>
> >> pages (values) and sets<br>
> >> > the reference back to null once entries are<br>
> >> going to be evicted<br>
> >> > (PageReference.setPage(null)).<br>
> >> ><br>
> >> > However, I think the whole strategy of<br>
> having to<br>
> >> have in-memory nodes<br>
> >> > might not be best. Maybe it's better to use<br>
> >> off-heap memory for the<br>
> >> > pages itself with MemorySegments, but the<br>
> pages<br>
> >> are not of a fixed<br>
> >> > size, thus it may get tricky.<br>
> >> ><br>
> >> > The test mentioned is this:<br>
> >> ><br>
> >><br>
> <a href="https://github.com/sirixdb/sirix/blob/248ab141632c94c6484a3069a056550516afb1d2/bundles/sirix-core/src/test/java/io/sirix/service/json/shredder/JsonShredderTest.java#L69" rel="noreferrer noreferrer" target="_blank">https://github.com/sirixdb/sirix/blob/248ab141632c94c6484a3069a056550516afb1d2/bundles/sirix-core/src/test/java/io/sirix/service/json/shredder/JsonShredderTest.java#L69</a> <<a href="https://urldefense.com/v3/__https://github.com/sirixdb/sirix/blob/248ab141632c94c6484a3069a056550516afb1d2/bundles/sirix-core/src/test/java/io/sirix/service/json/shredder/JsonShredderTest.java*L69__;Iw!!ACWV5N9M2RV99hQ!MWZDuvCBsbZSYul-HLDtF_j1IBD6osBF4cBVE_bg0yM5zCqYFwzLLp7nKN3b1hq1XVFRreqUVaXiKuXjUwGbxpjjFRtH2qmJ$" rel="noreferrer noreferrer" target="_blank">https://urldefense.com/v3/__https://github.com/sirixdb/sirix/blob/248ab141632c94c6484a3069a056550516afb1d2/bundles/sirix-core/src/test/java/io/sirix/service/json/shredder/JsonShredderTest.java*L69__;Iw!!ACWV5N9M2RV99hQ!MWZDuvCBsbZSYul-HLDtF_j1IBD6osBF4cBVE_bg0yM5zCqYFwzLLp7nKN3b1hq1XVFRreqUVaXiKuXjUwGbxpjjFRtH2qmJ$</a>> <<a href="https://github.com/sirixdb/sirix/blob/248ab141632c94c6484a3069a056550516afb1d2/bundles/sirix-core/src/test/java/io/sirix/service/json/shredder/JsonShredderTest.java#L69" rel="noreferrer noreferrer" target="_blank">https://github.com/sirixdb/sirix/blob/248ab141632c94c6484a3069a056550516afb1d2/bundles/sirix-core/src/test/java/io/sirix/service/json/shredder/JsonShredderTest.java#L69</a> <<a href="https://urldefense.com/v3/__https://github.com/sirixdb/sirix/blob/248ab141632c94c6484a3069a056550516afb1d2/bundles/sirix-core/src/test/java/io/sirix/service/json/shredder/JsonShredderTest.java*L69__;Iw!!ACWV5N9M2RV99hQ!MWZDuvCBsbZSYul-HLDtF_j1IBD6osBF4cBVE_bg0yM5zCqYFwzLLp7nKN3b1hq1XVFRreqUVaXiKuXjUwGbxpjjFRtH2qmJ$" rel="noreferrer noreferrer" target="_blank">https://urldefense.com/v3/__https://github.com/sirixdb/sirix/blob/248ab141632c94c6484a3069a056550516afb1d2/bundles/sirix-core/src/test/java/io/sirix/service/json/shredder/JsonShredderTest.java*L69__;Iw!!ACWV5N9M2RV99hQ!MWZDuvCBsbZSYul-HLDtF_j1IBD6osBF4cBVE_bg0yM5zCqYFwzLLp7nKN3b1hq1XVFRreqUVaXiKuXjUwGbxpjjFRtH2qmJ$</a>>><br>
> >> ><br>
> >> > I can upload the JSON file somewhere for a<br>
> >> couple of days if needed.<br>
> >> ><br>
> >> > Caused by: java.lang.OutOfMemoryError<br>
> >> > at<br>
> >> ><br>
> >> <br>
> java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)<br>
> >> > at<br>
> >> ><br>
> >> <br>
> java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)<br>
> >> > at<br>
> >> ><br>
> >> <br>
> java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)<br>
> >> > at<br>
> >> ><br>
> >> <br>
> java.base/java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:542)<br>
> >> > at<br>
> >> ><br>
> >> <br>
> java.base/java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:567)<br>
> >> > at<br>
> >> ><br>
> >> <br>
> java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:670)<br>
> >> > at<br>
> >> ><br>
> >> <br>
> java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)<br>
> >> > at<br>
> >> ><br>
> >> <br>
> java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174)<br>
> >> > at<br>
> >> ><br>
> >> <br>
> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)<br>
> >> > at<br>
> >> ><br>
> >> <br>
> java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)<br>
> >> > at<br>
> >> > <a href="http://io.sirix.access.trx.page" rel="noreferrer noreferrer" target="_blank">io.sirix.access.trx.page</a><br>
> <<a href="https://urldefense.com/v3/__http://io.sirix.access.trx.page__;!!ACWV5N9M2RV99hQ!MWZDuvCBsbZSYul-HLDtF_j1IBD6osBF4cBVE_bg0yM5zCqYFwzLLp7nKN3b1hq1XVFRreqUVaXiKuXjUwGbxpjjFeoArpQf$" rel="noreferrer noreferrer" target="_blank">https://urldefense.com/v3/__http://io.sirix.access.trx.page__;!!ACWV5N9M2RV99hQ!MWZDuvCBsbZSYul-HLDtF_j1IBD6osBF4cBVE_bg0yM5zCqYFwzLLp7nKN3b1hq1XVFRreqUVaXiKuXjUwGbxpjjFeoArpQf$</a>><br>
> >> <<a href="http://io.sirix.access.trx.page" rel="noreferrer noreferrer" target="_blank">http://io.sirix.access.trx.page</a><br>
> <<a href="https://urldefense.com/v3/__http://io.sirix.access.trx.page__;!!ACWV5N9M2RV99hQ!MWZDuvCBsbZSYul-HLDtF_j1IBD6osBF4cBVE_bg0yM5zCqYFwzLLp7nKN3b1hq1XVFRreqUVaXiKuXjUwGbxpjjFeoArpQf$" rel="noreferrer noreferrer" target="_blank">https://urldefense.com/v3/__http://io.sirix.access.trx.page__;!!ACWV5N9M2RV99hQ!MWZDuvCBsbZSYul-HLDtF_j1IBD6osBF4cBVE_bg0yM5zCqYFwzLLp7nKN3b1hq1XVFRreqUVaXiKuXjUwGbxpjjFeoArpQf$</a>>>.NodePageTrx.parallelSerializationOfKeyValuePages(NodePageTrx.java:442)<br>
> >> ><br>
> >> > I've uploaded several JFR recordings and logs<br>
> >> over here (maybe besides<br>
> >> > the async profiler JFR files the zgc-detailed<br>
> >> log is most interesting):<br>
> >> ><br>
> >> ><br>
> >> <a href="https://github.com/sirixdb/sirix/tree/main/bundles/sirix-core" rel="noreferrer noreferrer" target="_blank">https://github.com/sirixdb/sirix/tree/main/bundles/sirix-core</a><br>
> <<a href="https://urldefense.com/v3/__https://github.com/sirixdb/sirix/tree/main/bundles/sirix-core__;!!ACWV5N9M2RV99hQ!MWZDuvCBsbZSYul-HLDtF_j1IBD6osBF4cBVE_bg0yM5zCqYFwzLLp7nKN3b1hq1XVFRreqUVaXiKuXjUwGbxpjjFYBlqOOx$" rel="noreferrer noreferrer" target="_blank">https://urldefense.com/v3/__https://github.com/sirixdb/sirix/tree/main/bundles/sirix-core__;!!ACWV5N9M2RV99hQ!MWZDuvCBsbZSYul-HLDtF_j1IBD6osBF4cBVE_bg0yM5zCqYFwzLLp7nKN3b1hq1XVFRreqUVaXiKuXjUwGbxpjjFYBlqOOx$</a>> <<a href="https://github.com/sirixdb/sirix/tree/main/bundles/sirix-core" rel="noreferrer noreferrer" target="_blank">https://github.com/sirixdb/sirix/tree/main/bundles/sirix-core</a> <<a href="https://urldefense.com/v3/__https://github.com/sirixdb/sirix/tree/main/bundles/sirix-core__;!!ACWV5N9M2RV99hQ!MWZDuvCBsbZSYul-HLDtF_j1IBD6osBF4cBVE_bg0yM5zCqYFwzLLp7nKN3b1hq1XVFRreqUVaXiKuXjUwGbxpjjFYBlqOOx$" rel="noreferrer noreferrer" target="_blank">https://urldefense.com/v3/__https://github.com/sirixdb/sirix/tree/main/bundles/sirix-core__;!!ACWV5N9M2RV99hQ!MWZDuvCBsbZSYul-HLDtF_j1IBD6osBF4cBVE_bg0yM5zCqYFwzLLp7nKN3b1hq1XVFRreqUVaXiKuXjUwGbxpjjFYBlqOOx$</a>>><br>
> >> ><br>
> >> > kind regards<br>
> >> > Johannes<br>
> >><br>
> <br>
</blockquote></div>