[9] RFR(S): 8075214: SIGSEGV in nmethod sweeping

[Tobias Hartmann]

Hi,

please review the following patch.

https://bugs.openjdk.java.net/browse/JDK-8075214
http://cr.openjdk.java.net/~thartmann/8075214/webrev.00/

Problem:
The test uses the Whitebox API to enforce sweeping by creating and starting a 'CodeCacheSweeperThread'. During creation of the thread, the interpreter crashes in j.l.ThreadGroup.add(Thread t) [1] while executing a subtype check to validate that 't' is a subtype of j.l.Thread [2]. The problem is that we pass 'JavaThread->threadObj()' to 'ThreadGroup.add' which is invalid due to a GC that moved the object. The GC does not know about the thread because it was not yet added to the threads list and therefore does not update the oop.

Solution:
Instead of calling 'JavaThread::allocate_threadObj', the initialization is moved to the caller to make sure that setting the thread oop is done together with adding the thread to the threads list. I also fixed the missing oom handling described as one of the problems in JDK-8072377 [3].

Testing:
- 1k runs of failing testcase
- JPRT

Thanks,
Tobias

[1] http://hg.openjdk.java.net/jdk9/hs-comp/jdk/file/tip/src/java.base/share/classes/java/lang/ThreadGroup.java#l896
[2] see '__ gen_subtype_check' in 'TemplateTable::aastore'
[3] https://bugs.openjdk.java.net/browse/JDK-8072377