RFR: JDK-8265298: Hard VM crash when deadlock between "access" and higher ranked lock is detected [v2]

[Aleksey Shipilev]

> I stumbled upon this when doing some Shenandoah work. The development code tried to lock the `leaf` lock, while already holding the `access` lock. Normally it would have been detected by VM, but instead, we tried to recursively acquire `tty_lock` for `Thread::print_owned_locks`. But that `tty` lock is still ranked higher than `access`, so deadlock detection triggers over and over again until we run out of stack and crash hard. `tty` and `access` are ranked this way because of [JDK-8214315](https://bugs.openjdk.java.net/browse/JDK-8214315). Read the rest in the bug.  
> 
> I believe the way out is to only enter `Thread::print_owned_locks` when we know deadlock detection code would not run in circles. New test for `access` and `leaf` shows the original failure. Another new test checks `tty` and `special` to verify that the check should be `> tty`, not `>= tty`.
> 
> Additional testing:
>  - [x] Linux x86_64 fastdebug tier1
>  - [x] New regression tests (fail without the patch, pass wit it)

Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:

  Patricio's review

-------------

Changes:
  - all: https://git.openjdk.java.net/jdk/pull/3524/files
  - new: https://git.openjdk.java.net/jdk/pull/3524/files/ecf6e5bb..7c4410df

Webrevs:
 - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=3524&range=01
 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=3524&range=00-01

  Stats: 19 lines in 1 file changed: 5 ins; 5 del; 9 mod
  Patch: https://git.openjdk.java.net/jdk/pull/3524.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/3524/head:pull/3524

PR: https://git.openjdk.java.net/jdk/pull/3524