RFR 8252249: nsk/stress/stack/stack016.java fails with "Error: TEST_BUG: trickyRecursion() must throw an error anyway!"

[Harold Seigel]

Hi,

Please review this change to hotspot test 
vmTestbase/nsk/stress/stack/stack016.java.  The test calls a recursive 
method and keeps track of the number of repetitions needed to cause an 
exception.  It then runs a bunch of threads that call the recursive 
method for a multiple of the repetition number, expecting each of them 
to get a StackOverflowError or OutOfMemoryError exception.  
Occasionally, the test fails because one of the threads does not throw 
an exception.

This change tries to fix this in two ways.  One, by making sure that the 
thread used to determine the number of repetitions gets a 
StackOverflowError or OutOfMemoryError exception, and not some other 
unexpected exception.  The other way is to run the test twice, once with 
-Xcomp and once with -Xint, to ensure that thread stack consumption 
doesn't vary because the original thread called an interpreted method 
and a subsequent thread called a compiled method.

Open Webrev: 
http://cr.openjdk.java.net/~hseigel/bug_8252249.stack/webrev/index.html

JBS Bug: https://bugs.openjdk.java.net/browse/JDK-8252249

The modified test was tested on Mac OS, Linux x64, and Windows.

Thanks, Harold