RFR: JDK-8302618: [macOS] Problem typing uppercase letters with java.awt.Robot when moving mouse [v7]

[Alexey Ivanov]

On Fri, 21 Jul 2023 03:39:27 GMT, Sergey Bylokhov <serb at openjdk.org> wrote:

>> @mrserb 
>> 
>>> Just to clarify an idea of this change. We would like to maintain the "flag" keys pressed by the user and maintain the list of current flags. So if the user will press SHIFT key, we will use that modifier for all next keys until shift will be released.
>> 
>> That is correct.
>> 
>>> It looks similar to how the mouseMove is implemented in CRobot, where we maintain the mouseLastX/mouseLastY, and we ignore the current location of the mouse for clicks, but use the latest saved value.
>> 
>> 
>>> Probably we can do the same on the java side in this patch? it will simplify the native code a bit.
>> 
>> Yes, the code for mouseEvent looks similar to what we are trying to achieve for keyEvent but there are a few differences too-                 
>> - `mouseEvent(int lastX, int lastY, int buttonsState, boolean isButtonsDownState, boolean isMouseMove)`,  lastX  & lastY are already part of the parameters being sent from Java to native side.
>> 
>> - `keyEvent(int javaKeyCode, boolean keydown)` - Are you suggesting that a modifier flag state be sent from Java to native side? This would involve changing the present native `keyEvent` signature and adding code to native side to evaluate the new modifier flag state from Java and then map it to the corresponding [kCGEventFlagMask](https://developer.apple.com/documentation/coregraphics/cgeventflags/kcgeventflagmaskalphashift?language=objc).
>> 
>> Having everything on native side seems to be easier because with the present fix, the equivalent modifier flag mask is obtained using javaKeyCode + key state and a quick `NSDictionary modifierKeyToMaskMap` lookup without the need of adding or changing  keyEvent signature. 
>> 
>>> BTW do we need to set all this modifiers for the mouse clicks as well?
>> 
>> I don't think it is applicable for mouse clicks, as the issue (details below) occurs only when flagState is obtained using - `CGEventSourceFlagsState(kCGEventSourceStateHIDSystemState)` which was the case for keyEvents. I can double-check on this again.
>> 
>> **Issue:** 
>> _With the original code, the issue occurs at [CRobot.m#L295](https://github.com/openjdk/jdk/blob/ac6af6a64099c182e982a0a718bc1b780cef616e/src/java.desktop/macosx/native/libawt_lwawt/awt/CRobot.m#L295.) The flags gets reset or cleared when mouse is moved physically in unison with Robot's key events. The physical mouse movement causes the event flags to be reset._
>
>>I don't think it is applicable for mouse clicks, as the issue (details below) occurs only when flagState is obtained using - CGEventSourceFlagsState(kCGEventSourceStateHIDSystemState) which was the case for keyEvents. I can double-check on this again.
> 
> yes, please.
> if the physical mouse movement causes the "event flags" to be reset, then what about  "event flags" for mouse events

> I see your point. I think based on what @mrserb and you have discussed, I see 3 cases here:
> 
> 1. Robot mouse events + external manual mouse move- In this case user mouse events will cause the mouse to be at wrong position thus causing erroneous result as expected. And this is not a valid use case.

What I mean is to look further whether this situation affects any internal state of Robot so that wrong mouse events are posted.

It could still be a possible scenario. Consider Robot is moving mouse slowly, the user presses a mouse button; the following events generated by the robot should indicate the mouse button is pressed until the user releases the mouse button.

There's also the **fourth possibility**: Robot generates keyboard events while the user presses modifier keys on the keyboard. It's an invalid scenario for tests but not impossible. Should the robot combine its internal state with the global hardware state?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14744#issuecomment-1648461865