RFR 8204525: [TESTBUG] runtime/NMT/MallocStressTest.java ran out of java heap

[Harold David Seigel]

Hi,

Please review this small change to fix test bug JDK-8204525. Test 
MallocStressTest quickly allocates lots of malloc memory and slowly 
releases it.  It checks that NMT properly measures the amount of 
allocated memory.  The test occasionally fails when it runs out of Java 
heap before the malloc memory allocations completed.

The test has several threads that loop, allocating lots of memory.  A 
thread's allocations stop when its malloc allocation fails, or when a 
certain amount of time expires.  This fix also stops a thread's 
allocation when an OutOfMemoryError exception gets thrown.

Open Webrev: http://cr.openjdk.java.net/~hseigel/bug_8204525/webrev/

JBS Bug:  https://bugs.openjdk.java.net/browse/JDK-8204525

The fix was tested on Linux X64, Windows, Solaris, and Mac.  It was also 
tested on Linux X64 with -Xmx20M, which, before this fix, reliably 
reproduced the failure.

Thanks, Harold