RFR: JDK-8302618: [macOS] Problem typing uppercase letters with java.awt.Robot when moving mouse [v7]

[Harshitha Onkar]

On Fri, 21 Jul 2023 03:39:27 GMT, Sergey Bylokhov <serb at openjdk.org> wrote:

>> @mrserb 
>> 
>>> Just to clarify an idea of this change. We would like to maintain the "flag" keys pressed by the user and maintain the list of current flags. So if the user will press SHIFT key, we will use that modifier for all next keys until shift will be released.
>> 
>> That is correct.
>> 
>>> It looks similar to how the mouseMove is implemented in CRobot, where we maintain the mouseLastX/mouseLastY, and we ignore the current location of the mouse for clicks, but use the latest saved value.
>> 
>> 
>>> Probably we can do the same on the java side in this patch? it will simplify the native code a bit.
>> 
>> Yes, the code for mouseEvent looks similar to what we are trying to achieve for keyEvent but there are a few differences too-                 
>> - `mouseEvent(int lastX, int lastY, int buttonsState, boolean isButtonsDownState, boolean isMouseMove)`,  lastX  & lastY are already part of the parameters being sent from Java to native side.
>> 
>> - `keyEvent(int javaKeyCode, boolean keydown)` - Are you suggesting that a modifier flag state be sent from Java to native side? This would involve changing the present native `keyEvent` signature and adding code to native side to evaluate the new modifier flag state from Java and then map it to the corresponding [kCGEventFlagMask](https://developer.apple.com/documentation/coregraphics/cgeventflags/kcgeventflagmaskalphashift?language=objc).
>> 
>> Having everything on native side seems to be easier because with the present fix, the equivalent modifier flag mask is obtained using javaKeyCode + key state and a quick `NSDictionary modifierKeyToMaskMap` lookup without the need of adding or changing  keyEvent signature. 
>> 
>>> BTW do we need to set all this modifiers for the mouse clicks as well?
>> 
>> I don't think it is applicable for mouse clicks, as the issue (details below) occurs only when flagState is obtained using - `CGEventSourceFlagsState(kCGEventSourceStateHIDSystemState)` which was the case for keyEvents. I can double-check on this again.
>> 
>> **Issue:** 
>> _With the original code, the issue occurs at [CRobot.m#L295](https://github.com/openjdk/jdk/blob/ac6af6a64099c182e982a0a718bc1b780cef616e/src/java.desktop/macosx/native/libawt_lwawt/awt/CRobot.m#L295.) The flags gets reset or cleared when mouse is moved physically in unison with Robot's key events. The physical mouse movement causes the event flags to be reset._
>
>>I don't think it is applicable for mouse clicks, as the issue (details below) occurs only when flagState is obtained using - CGEventSourceFlagsState(kCGEventSourceStateHIDSystemState) which was the case for keyEvents. I can double-check on this again.
> 
> yes, please.
> if the physical mouse movement causes the "event flags" to be reset, then what about  "event flags" for mouse events

@mrserb 

> if the physical mouse movement causes the "event flags" to be reset, then what about "event flags" for mouse events

Robot's mouse events are handled differently in comparison to keyEvents , mouse events are processed by going through `CRobot_mouseEvent` > `HandleRobotButton`  > `PostMouseEvent` within CRobot.m. Current flag state of the event source is never queried for mouseEvents, so the issue of resetting of event flag does not occur. Moreover `initFlags` which is queried in initRobot is used within `CRobot_keyEvent` only.

Additionally, I believe concurrent external manual mouse movement is not supported when Robot is performing mouse events. This would produce erroneous result. For example, running MouseLocationOnScreen.java test or any location related test with Robot + simultaneous external mouse movement will fail. So the question of handling physical external mouse movements during Robot's mouse events looks to be invalid or not applicable here.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14744#issuecomment-1646242299