RFR: 8283710: JVMTI: Use BitSet for object marking [v8]

[Roman Kennke]

> JVMTI heap walking marks objects in order to track which have been visited already. In order to do that, it uses bits in the object header. Those are the same bits that are also used by some GCs to mark objects (the lowest two bits, also used by locking code). Some GCs also use the bits in order to indicate 'forwarded' objects, where the upper bits of the header represent the forward-pointer. In the case of Shenandoah, it's even more problematic because this happens concurrently, even while JVMTI heap walks can intercept. So far we carefully worked around that problem, but it becomes very problematic in Lilliput, where accesses to the Klass* also requires to decode the header, and figure out what bits means what.
> 
> In addition to that, marking objects in their header requires that the original header gets saved and restored. We only do that for 'interesting' headers, that is headers that have a stack-lock, monitor or hash-code. All other headers are reset to their default value. This means we are losing object's GC age. This is not catastrophic, but nontheless interferes with GC. 
> 
> JFR already has a datastructure called BitSet to support object marking without messing with object's headers. We can use that in JVMTI too.
> 
> Testing:
>  - [x] tier1
>  - [x] tier2
>  - [x] tier3
>  - [x] serviceability/jvmti
>  - [x] vmTestbase/nsk/jvmti

Roman Kennke has updated the pull request incrementally with one additional commit since the last revision:

  Add some basic tests for ObjectBitSet

-------------

Changes:
  - all: https://git.openjdk.java.net/jdk/pull/7964/files
  - new: https://git.openjdk.java.net/jdk/pull/7964/files/9ea51d78..b2d913b9

Webrevs:
 - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=7964&range=07
 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=7964&range=06-07

  Stats: 66 lines in 1 file changed: 66 ins; 0 del; 0 mod
  Patch: https://git.openjdk.java.net/jdk/pull/7964.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/7964/head:pull/7964

PR: https://git.openjdk.java.net/jdk/pull/7964