Request for Backport of JDK-8159127 hprof heap dumps broken for lambda classdata

[Basil Crow]

Hi all,

I am writing to request a backport of JDK-8159127 to 8u. My high-level
goal is to extract a usable hprof heap from a core file of a Java
program where lambdas are in use. By "usable" I mean that finding
objects by retained size in VisualVM should work. This works in 9u and
later, but it does not work in 8u due to JDK-8159127. Note that a
usable hprof heap dump is generated in 8u when a live process (rather
than a core file) is used or when lambdas are not in use, as described
in JDK-8159127. This bug effectively prevents one from debugging
memory leaks from core dumps on 8u.

Unfortunately, I don't think the backport will be trivial. I applied
the fixes from JDK-8159127 onto 8u. Apart from file renames and a
trivial merge conflict in vmStructs.cpp, I was able to apply all the
changes and build a JDK. All existing automated tests still passed,
but the new automated test added in JDK-8159127 did not run for some
reason. I then tried to verify the fix with a manual test (described
below), which failed. It is clear that further work is required for
this backport. As someone who just built OpenJDK for the first time
minutes ago, I am likely not the best candidate to do this work.

For anyone who is interested in taking this on, here is the procedure
I used for manual testing. This is the same procedure I described in
JDK-8159127, but with some additional clarifications.

First, build and run the test program:

$ cat LambdaSleep.java
import java.util.Arrays;
public class LambdaSleep {
  public static void main(String[] args) throws InterruptedException {
    String[] words = new String[] {"longer", "short"};
    Arrays.sort(words, (first, second) ->
Integer.compare(first.length(), second.length()));
    Thread.sleep(300000);
  }
}

$ javac LambdaSleep.java
$ java LambdaSleep &
[1] 4528

Next, we'll take two heap dumps of this program. The first,
live4528.hprof, comes from the live process:

$ jmap -dump:format=b,file=live4528.hprof 4528
Heap dump file created

The second, core4528.hprof, comes from a core dump. First, let's take
the core dump:

# gcore 4528
[...]
Saved corefile core.4528
[Inferior 1 (process 4528) detached]

Next, let's generate an hprof heap dump from the core dump. In 9u and
later, this is done with jhsdb(1):

$ jhsdb jmap --binaryheap --dumpfile core4528.hprof --exe
$JAVA_HOME/bin/java --core core.4528
Attaching to core core.4528 from executable
/opt/jdk-11.0.3+7/bin/java, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 11.0.3+7
heap written to core4528.hprof

In 8u, this is done with jmap(1):

$ jmap -dump:format=b,file=core4528.hprof /opt/jdk/bin/java core.4528
Attaching to core core.4528 from executable /opt/jdk/bin/java, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.60-b23
Dumping heap to core4528.hprof ...
Heap dump file created

Either way, we now have two hprof heap dumps, which should be
functionally identical when used in VisualVM.

Next, open each hprof heap dump in VisualVM and attempt to find
objects by retained size. For the heap dump generated from the live
process, this will work for all Java versions. For the heap dump
generated from the core file, this will only work in 9u and later, due
to JDK-8159127. On 8u, VisualVM will hang on "computing retained
sizes." After applying the patches from JDK-8159127 onto 8u, the hang
no longer occurred, but VisualVM threw a NullPointerException instead.
To complete this backport, this should be investigated. It also
remains unclear why the automated test didn't run on 8u.

Note that Eclipse MAT prior to 1.6.1 also used to crash when presented
with these bad hprof heap dumps. However, as of 1.6.1, a workaround
[1] was put in place that substitutes dummy classes for the missing
information from the bad hprof heap dump. Therefore I recommend
testing with VisualVM, which still fails explicitly when a bad hprof
heap dump is encountered.

Thanks in advance from the production debugging trenches!
Basil

[1] https://bugs.eclipse.org/bugs/show_bug.cgi?id=471757