RFR [jdk11] : 8242174: [macos] The NestedModelessDialogTest test make the macOS unstable

[Baesken, Matthias]

Hello, please  review the backport of  8242174  to jdk11 .
The patch mostly applies cleanly to jdk11 , but  in test/jdk/ProblemList.txt  we have a small difference so manual intervention has needed.


Bug / webrev jdk11 :

https://bugs.openjdk.java.net/browse/JDK-8242174

http://cr.openjdk.java.net/~mbaesken/webrevs/8242174_jdk11_0/

Webrev from jdk/jdk :

https://hg.openjdk.java.net/jdk/jdk/rev/d5d7e0df08d2

Thanks, Matthias