<AWT Dev> Bug connected to sun.awt.datatransfer.DataTransferer on OS X

[Hendrik Schreiber]

Hi,

as already discussed in thread http://mail.openjdk.java.net/pipermail/macosx-port-dev/2014-August/006706.html, there is a bug connected to sun.awt.datatransfer.DataTransferer that appears at least on OS X.

For OS X the problem is, that the method sun.awt.datatransfer.DataTransferer#convertData(...) can be called in a way that leads to unbalanced calls of 

getToolkitThreadBlockedHandler().enter()

and

getToolkitThreadBlockedHandler().exit()

In essence, if exit() is called twice in a row, we crash. This is IMO due to unfortunate scheduling of the OS X AppKit and Java EDT and the fact that currently, on OS X the secondary AWT loop can only be started once properly.

With the current code, the following scenario (or something with the same effect) is possible:

AppKit: convertData()
AppKit: executeOnEventHandlerThread()
AppKit: lock()
AppKit: enter() [unlock(), startAWTLoop]
{now in secondary loop, but still the same thread}
   AppKit: convertData()
   AppKit: executeOnEventHandlerThread()
   AppKit: lock()
   AppKit: enter() [unlock(), doAWTRunLoop]
EDT: lock()
EDT: exit() [stopAWTRunLoop]
EDT: unlock()
EDT: lock()
EDT: exit() [stopAWTRunLoop] <= CRASH!!
EDT: unlock()

Explanation: convertData() is called on DataTransferer, lock(), unlock(), enter() and exit() on sun.lwawt.macosx.CToolkitThreadBlockedHandler.
In CToolkitThreadBlockedHandler we call LWCToolkit.doAWTRunLoop() and LWCToolkit.stopAWTRunLoop().

I wrote a simple test case that calls convertData() repeatedly from the AppKit thread and produces the crash reliably. You can get it as a (cumbersome) Maven project from http://www.beatunes.com/download/drag_crash.zip (may not be the prettiest code, but you get the gist of it). Alternatively, just download http://www.beatunes.com/download/drag_crash-java-0.9.0-SNAPSHOT.jar and http://www.beatunes.com/download/libdrag_crash.jnilib and start the class com.tagtraum.dragcrash.DragCrash

Since DataTransferer is shared, a fix may affect all platforms.

On OS X, one can basically start another secondary AWTRunLoop on top of the current one and so on. However, the current enter()/exit() code in DataTransfer was clearly written with the intent of a balance: I.e. an enter() call is followed by an exit() call.
This works, unless we for some reason call exit() twice in a row, which is possible as shown above. Then at least the code for OS X breaks, as CToolkitThreadBlockedHandler does not stack its awtRunLoopMediators instances. Not sure how other platforms behave here.

To fix this, one could perhaps build a stack into CToolkitThreadBlockedHandler.
Or one could try to prevent the AppKit thread from entering convertData multiple times in order to not run into this problem in the first place.

But this does not necessarily work for other platforms. As a matter of fact - I don't know whether this is even an issue for other platforms. I simply don't know enough about the whole mechanism/design to make a good judgement call.

Perhaps someone with deeper knowledge about how this was intended to work on all platforms, can comment?

Cheers,

-hendrik