<Swing Dev> JDK-8189938: Proposed patch

[Matthias Bläsing]

Hi all,

last month I saw the message from Reinhard Pointer on this list:

http://mail.openjdk.java.net/pipermail/swing-dev/2018-January/008132.html

referring to this bug:

https://bugs.openjdk.java.net/browse/JDK-8189938

With the instructions given in:

https://bugs.openjdk.java.net/browse/JDK-8193928 (closed as duplicate of JDK-8189938)

I was able to reproduce the problem on a clean Windows 10 Installation,
as described in the initial report.

The resulting exception leads to my proposed fix:

        Could not initialize COM: HRESULT=0x80010106  

That HRESULT translates to: RPC_E_CHANGED_MODE

This means, that the COM subsystem is already initialized, but with a
different mode. The code uses CoInitialize and this initializes the
threading module to be apartment threaded. The above HRESULT means the
thread was already initialized to be multi-threaded.

The mode can't be changed after is was set once, so instead of bailing
out, I suggest to accept the situation and initialize to be multi-
threaded.

The consequence is that calls from COM to Java could end up on the
wrong thread. A quick look through the code suggest, that COM advises
are not used, so this is a risk that should be taken.

The fix in this case works like this:

 * try to initialize COM as it was
 * check the return
 * if it is RPC_E_CHANGED_MODE, retry initialization als multi-threaded
 * only if that fails raise an exception

One could argue, that if COM is already initialized, you don't need to
do it yourself, but documentation states, that every successful call to
CoInitialize/CoInitializeEx must be paired with CoUninitialize.

D3DPipelineManager tracks this in bComInitialized, the other two
callers in the JDK are:

- PLATFORM_API_WinOS_DirectSound.cpp
- ShellFolder2.cpp

Both are covered in the patches. For ShellFolder2 there is an
initialization check, this is modified as described above. And if both
initialization variant (apartment threaded and multi threaded fail)
this raises an exception.

PLATFORM_API_WinOS_DirectSound is slightly different. The fallback in
the initialization is still done, but no exception is raised in the
error case. This follows the code flow before this change.

I attached the diffs against JDK10 and JDK9 to this email. If they are
stripped, they can be found here:

http://www.doppel-helix.eu/JDK8189938-openjdk9.diff
http://www.doppel-helix.eu/JDK8189938-openjdk10.diff

I hope this helps with a fix.

Greetings

Matthias
-------------- next part --------------
A non-text attachment was scrubbed...
Name: JDK8189938-openjdk9.diff
Type: text/x-patch
Size: 2370 bytes
Desc: not available
URL: <http://mail.openjdk.java.net/pipermail/swing-dev/attachments/20180208/5d765858/JDK8189938-openjdk9.diff>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: JDK8189938-openjdk10.diff
Type: text/x-patch
Size: 2541 bytes
Desc: not available
URL: <http://mail.openjdk.java.net/pipermail/swing-dev/attachments/20180208/5d765858/JDK8189938-openjdk10.diff>