RFR (XS): 8202049: G1: ReferenceProcessor doesn't handle mark stack overflow

[Thomas Schatzl]

Hi all,

  can I have reviews for this change that makes sure that if we get a
mark stack overflow during reference processing in the Remark pause we
error out in the VM because we can't recover from that.

I.e. recovery would basically mean restarting from scratch, but
visiting Reference objects multiple times can have lots of side-
effects.

So the only current option is to give up with a "nice" error message
for now.

Note that there is no known actual problem because of this, i.e.
overflow during remark is (supposedly) really rare.

I spent some time trying to create a reproducer, but failed. I will
file an extra RFE for this.

CR:
https://bugs.openjdk.java.net/browse/JDK-8202049
Webrev:
http://cr.openjdk.java.net/~tschatzl/8202049/webrev
Testing:
running this patch for months in my usual testing routine for other
patches without any error.

Thanks,
  Thomas