[11] Review request: 8087581: Headless Monocle causes core dump when rendering javafx.scene.text.Text

[Alex Kashchenko]

Hi OpenJFX developers,

issue: https://bugs.openjdk.java.net/browse/JDK-8087581
webrev: http://cr.openjdk.java.net/~akasko/ojfx/8087581/webrev.01/

I am working on windows builds at Red Hat, and some time ago stumbled 
upon the windows-only font-related crash inside directwrite.cpp. Crash 
was 100% reproducible (but only with openjdk8+openjfx - not with Oracle 
jdk) with VocabHunter [1] project - running:

gradlew :gui:test

Problem appeared to be the same as logged in JDK-8087581. It was 
narrowed down to this call [2].

Creating COM object instance and immediately un-initializing COM seemed 
to be a wrong approach. Looking how COM is used in similar manner 
(singleton instance accessed from a single thread) in other projects 
[3][4], it appeared that there CoUninitialize either called on thread 
exit or not called at all.

In the proposed patch CoUninitialize is tied to this cleanup call [5].


Please note some possible problems with this patch, that may prevent it 
from being accepted:

  - patch is tested only with openjdk-8u161 + openjfx-8u161
  - I was unable to reproduce the problem with Oracle jdk
  - reproducer inside JDK-8087581 seems to be broken with recent jdk8 
(something is wrong with headless monocle lib [6] ), I used only 
VocabHunter project as a reproducer
  - the latest openjfx build I have is from openjfx/10/rt/ repo - do not 
have 11 (or dev) builds yet
  - VocabHunter cannot be built with openjfx-10, so the gradle command 
above won't work with it
  - I bundled reproducer as a standalone JAR, but it is still not a real 
reproducer, as that JAR is 60MB (with guice and other libs inside) and 
won't run on openjfx-10 (again some headless monocle incompatibility)
  - fontFactory is changed to "protected" here [7], to call the 
dispose() on it directly from D3DPipeline; it may be better to call 
dispose() for any fontFactory from GraphicsPipeline (parent) , but in 
testing it appeared to be, that parent's dispose() is never called from 
D3DPipeline.dispose(), as nDispose() [8] call inside it never exits 
(process exits first); thus in patch fontFactory.dispose() is called 
before the nDispose() call


[1] https://github.com/VocabHunter/VocabHunter
[2] http://hg.openjdk.java.net/openjfx/jfx-dev/rt/rev/86a1b36090e4#l22.599
[3] https://hg.mozilla.org/mozilla-central/rev/a67425aa4728#l1.44
[4] 
https://github.com/Microsoft/Windows-driver-samples/blob/master/print/XpsRasFilter/src/BitmapHandler.cpp#L64
[5] 
http://hg.openjdk.java.net/openjfx/jfx-dev/rt/file/375aedc5702a/modules/javafx.graphics/src/main/java/com/sun/javafx/tk/quantum/QuantumRenderer.java#l127
[6] http://mvnrepository.com/artifact/org.testfx/openjfx-monocle
[7] 
http://hg.openjdk.java.net/openjfx/jfx-dev/rt/file/375aedc5702a/modules/javafx.graphics/src/main/java/com/sun/prism/GraphicsPipeline.java#l58
[8] 
http://hg.openjdk.java.net/openjfx/jfx-dev/rt/file/375aedc5702a/modules/javafx.graphics/src/main/java/com/sun/prism/d3d/D3DPipeline.java#l164


-- 
-Alex