g1: remembered set scan costs very high w/ LRU cache type of behavior

[Peter Schuller]

> The problem you're seeing may be addressed by some
> work in progress that is being done under CR
>
> 6923991: G1: improve scalability of RSet scanning
>
> I don't think that work has been released yet.

Well, if this commit constitutes the work:

   http://hg.openjdk.java.net/jdk7/hotspot/hotspot/rev/0414c1049f15

Then I suspect I have it, because the JDK comes from:

   jdk-7-ea-bin-b84-linux-x64-18_feb_2010.bin

Which is 7 days after the commit happened.

I'll see about building from source to maks sure.

However, reading about it my initial impression that it is probably
not the same issue as it seems to be about many-core scalability of
the RS scanning. In my case I see poor performance even with a single
core, and to the extent it is a scaling issue I get the impression it
is a function of the number of RS tracked writes rather than the
number of CPU cores.

> If it hasn't,
> with your permission, we'll try your test program with
> it and see how well it does.

Of course.

Btw, I discovered something else. I have previously been doing some
testing on JDK:s build from the jdk 1.7 bsdport on FreeBSD and had
stability issues where the JVM would suddenly exit with no message
when I ran my clojure persistent LRU test with G1. I just hit this
behavior on the Linux host earlier today, with the snapshot binary
mentioned above, which means I can no longer chalk it up to a FreeBSD
specific issue.

I doubt I can do much to narrow it down to a very small test case, but
I could tidy up the test case a bit and make that available, and
perhaps produce an executable .jar with clojure + the test case if
that helps. I suspect (based on speculation rather than evidence) that
the allocation and write patterns being generated by the use of
persistent/immutable data structures is key to triggering it.

That is, unless there is an actual System.exit() somewhere in the
codebase... but I highly doubt there would be code that does this as a
function of some behavior which depends on the garbage collector
choice.

-- 
/ Peter Schuller