RFR: 8278020: ~13% variation in Renaissance-Scrabble [v2]

[Ioi Lam]

> We found that when CDS is enabled, there is a ~13% variation in the Renaissance-Scrabble benchmark between different builds of the JDK. In one example, only two core-lib classes, unrelated to the benchmark, changed between two builds, but one build is consistently faster than the other.
> 
> When CDS is disabled, we do not see such variations.
> 
> In the slow case, there seems to be frequent dcache misses when loading the `Klass::_vtable_len` field, which is at offset 24 from the beginning of the Klass (see [bug report](https://bugs.openjdk.java.net/browse/JDK-8278020) for details). 
> 
> We suspect that the problem is with the layout of the CDS archive. Specifically, in CDS, Klass objects are inter-mixed with other metadata objects (such as Methods). In contrast, when CDS is disabled, (on 64-bit platforms with compressed klass pointers), Klass objects are allocated in their own space, separated from other metadata objects.
> 
> My theory is: when CDS is enabled, perhaps the modification of an object that sits immediately above the Klass invalidates the cacheline that holds `Klass::_vtable_len`. In a different JDK build, the exact addresses of the metadata objects in the CDS archive may be slightly nudged so we don't see the cacheline effect anymore.
> 
> As an experiment, I swapped `Klass::_vtable_len` with `Klass::_modifier_flags` (which was at offset 164 before this patch), and the variation stopped. Both fields are 32 bits in size.
> 
> I have no concrete proof that my theory is correct, but this change seems to be harmless. @ericcaspole has run all the benchmarks in Oracle's CI and found consistent improvement with Renaissance-Scrabble, and no degradation in other benchmarks.

Ioi Lam has updated the pull request incrementally with one additional commit since the last revision:

  added comments about the location of vtable_len

-------------

Changes:
  - all: https://git.openjdk.java.net/jdk/pull/6838/files
  - new: https://git.openjdk.java.net/jdk/pull/6838/files/fd1318a3..55e71805

Webrevs:
 - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=6838&range=01
 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=6838&range=00-01

  Stats: 4 lines in 1 file changed: 3 ins; 0 del; 1 mod
  Patch: https://git.openjdk.java.net/jdk/pull/6838.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/6838/head:pull/6838

PR: https://git.openjdk.java.net/jdk/pull/6838