RFR: 8293872: Make runtime/Thread/ThreadCountLimit.java more robust

[Aleksey Shipilev]

I am seeing this failure frequently on linux-x86:


$ CONF=linux-x86-server-fastdebug make test TEST=runtime/Thread/ThreadCountLimit.java
...
stdout:
[1.852s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 320k, guardsize: 0k, detached.
[1.852s][warning][os,thread] Failed to start the native thread for java.lang.Thread "Thread-8246"
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 8388608 bytes for Failed to reserve memory for metaspace
# An error report file with more information is saved as:
# /home/shade/trunks/jdk/build/linux-x86-server-fastdebug/test-support/jtreg_test_hotspot_jtreg_runtime_Thread_ThreadCountLimit_java/scratch/0/hs_err_pid2107279.log
[2.019s][warning][os ] Loading hsdis library failed


It usually crashes when it tries to do heavy stuff in `OutOfMemoryError` block, which ends up either allocating the metadata for string concat method handles (the most frequent case), or allocating memory for resulting Strings. Sometimes even the threads themselves end up receiving a secondary OOM after main loop catches it.

I think we can strengthen this test significantly, by avoiding unnecessary things in dangerous loops.

Additionally, since the test can foobar the VM and/or experience heap memory runaway, it needs to run with othervm and fixed heap size.

After [JDK-8255450](https://bugs.openjdk.org/browse/JDK-8255450), this test still runs in tier2.

Additional testing:
 - [x] Linux x86_32 fastdebug, affected test in 100 repetitions (all passed!)
 - [ ]  Linux x86_64 fastdebug, affected test in 100 repetitions

-------------

Commit messages:
 - Fix

Changes: https://git.openjdk.org/jdk/pull/10290/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=10290&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8293872
  Stats: 49 lines in 1 file changed: 21 ins; 18 del; 10 mod
  Patch: https://git.openjdk.org/jdk/pull/10290.diff
  Fetch: git fetch https://git.openjdk.org/jdk pull/10290/head:pull/10290

PR: https://git.openjdk.org/jdk/pull/10290