RFR: 8270269: Desktop.browse method fails if earlier CoInitialize call as COINIT_MULTITHREADED [v3]

[Alexey Ivanov]

On Fri, 8 Dec 2023 17:12:29 GMT, Sergey Bylokhov <serb at openjdk.org> wrote:

>> The fix for a regression caused by the https://bugs.openjdk.org/browse/JDK-6508941. it does not take into account RPC_E_CHANGED_MODE when COM was already initialized using COINIT_MULTITHREADED mode.
>> 
>> @aivanov-jdk please take a look.
>
> Sergey Bylokhov has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Update src/java.desktop/windows/native/libawt/windows/awt_Desktop.cpp
>   
>   date
>   
>   Co-authored-by: Alexey Ivanov <alexey.ivanov at oracle.com>

> > The confusion could come from #14898 where [#14898 (comment)](https://github.com/openjdk/jdk/pull/14898#issuecomment-1748967705) us from calling DirectSound API which supports both MTA and STA.
> 
> Would like to clarify difference between this and that patch: this bug shows that some api does not support MTA mode, and intentionally requests the STA - so our code was broken by some external call -> enable COM+MTA.
>
> At the same time the [patch](https://github.com/openjdk/jdk/pull/14898) may change the current mode of the thread from STA to MTA and that seems could break all code executed after?

How come? It can't change: once COM is initialised on a thread, its threading model cannot be changed. This is what `RPC_E_CHANGED_MODE` error code conveys, in other words you request initialising COM in a threading model that's different the model COM has been initialised previously.

The code in #14898 always initialises COM in MTA. If COM is initialised in STA, the thread and COM remains in STA. The only difference is that it is still *valid* to call the DirectSound API because the API supports both MTA and STA modes.

This case corresponds to option 3 in [this comment](https://github.com/openjdk/jdk/pull/14898#issuecomment-1748967705).

In other words the workflow in case 3 of JDK-8301846 / PR 14898:


// some app code
COM -> STA = S_OK
// more code
AudioSystem.getMixerInfo():
    COM -> MTA = RPC_E_CHANGED_MODE
    // threading model still STA
    DirectSoundEnumerateW / DirectSoundCaptureEnumerateW
    skip ::CoUninitialize()
// thread continues with COM in STA


(I was even inclined to use [the same approach](https://github.com/openjdk/jdk/pull/14898#discussion_r1276539431) with a worker thread which initialises COM to avoid dealing with `RPC_E_CHANGED_MODE`. It was only after I had received [clarification from Microsoft](https://github.com/openjdk/jdk/pull/14898#issuecomment-1721687912) that I changed my opinion because (a) Initialising and uninitialising COM aren't heavy; (b) DirectSound APIs support both MTA and STA.)

-------------

PR Comment: https://git.openjdk.org/jdk/pull/17010#issuecomment-1850737151