RFR (S): Parallelize code cache scan better

[Aleksey Shipilev]

Hi,

Turns out, our parallel code cache iterator is not that parallel: it processes
entire code heaps in one stride. In our current mode, there are only three code
heaps. So, in some workloads, e.g. Compiler.sunflow, code cache scan for the
largest code heap for initial evacuation dominates the final mark pause time.

The fix is to parallelize iterator better:
  http://cr.openjdk.java.net/~shade/shenandoah/code-cache-parallel/webrev.01/

Before:

 Final Mark Pauses (N) = 0.35 s (a = 6797 us) (n =    51)
     (lvls, us = 4219, 5508, 6680, 7168, 12895)

    Initial Evacuation = 0.28 s (a = 5499 us) (n =    51)
     (lvls, us = 3516, 4570, 5625, 5996, 9967)

After:

 Final Mark Pauses (N) = 0.26 s (a = 5261 us) (n =    49)
     (lvls, us = 3574, 3926, 4375, 5547, 14308)

    Initial Evacuation = 0.16 s (a = 3183 us) (n =    49)
     (lvls, us = 2324, 2617, 2949, 3262, 5801)

Testing: hotspot_gc_shenandoah, jcstress -m quick

Thanks,
-Aleksey