[8u20] RFR 8038939: Some options related to RTM locking optimization works inconsistently

[Vladimir Kozlov]

Please, review 8u20 backport.

These changes were pushed into jdk9 today. They were tested by SQE using 
RTM tests (in review to be added to Hotspot jtreg tests). Current 
Nightly testing will show nothing because RTM optimization is off by 
default.

Changes from jdk9 applied to 8u without conflicts.

http://hg.openjdk.java.net/jdk9/hs-comp/hotspot/rev/ee2e2f0fe8bb
http://cr.openjdk.java.net/~kvn/8038939/webrev
https://bugs.openjdk.java.net/browse/JDK-8038939

Thanks,
Vladimir