Failed assert during VM_HeapWalkOperation

[Rodrigo Bruno]

Dear Shenandoah dev list,

my name is Rodrigo, I am an ETH postdoc researcher, and my group currently
has a project that involves using Shenandoah and tracing objects upon
request by Java code (yes, many people will think this is a terrible
idea... :-), I can explain if someone is interested...).

We are using a slightly modified copy of the VM_HeapWalkOperation VM
operation to trace objects starting for a particular initial object.
However we started noticing failed asserts as soon as we introduced more
load into the system (more allocations, leading to more GC effort).

After some debugging, it turns out to be a load barrier check that fails
during the VM_HeapWalkOperation because it i) sets the mark bits and ii)
tries to resolve oops (triggering the barrier). For some reason, the gc
state indicates that object headers should have the forwardee pointer in
place and thus it fails. I saw that there is already an exception for this
particular VM op but it does not cover this path.

This failed assert can be reproduced using an unmodified JDK. I just tested
it today by downloading [1], building, and creating a simple test which
basically calls a JNI function that calls JVMTI FollowReferences. I can
provide the sources to reproduce the failed assert. If I
use -XX:ShenandoahGCMode=passive, no barriers are enabled and everything
works fine. Dump file is attached.

I think this might be a bug. Please let me know if you need any further
information/help.

cheers,
rodrigo


[1] http://hg.openjdk.java.net/shenandoah/jdk11
-- 
rodrigo-bruno.github.io