RFR: JDK-8302618: [macOS] Problem typing uppercase letters with java.awt.Robot when moving mouse [v7]

[Harshitha Onkar]

On Fri, 21 Jul 2023 03:39:27 GMT, Sergey Bylokhov <serb at openjdk.org> wrote:

>> @mrserb 
>> 
>>> Just to clarify an idea of this change. We would like to maintain the "flag" keys pressed by the user and maintain the list of current flags. So if the user will press SHIFT key, we will use that modifier for all next keys until shift will be released.
>> 
>> That is correct.
>> 
>>> It looks similar to how the mouseMove is implemented in CRobot, where we maintain the mouseLastX/mouseLastY, and we ignore the current location of the mouse for clicks, but use the latest saved value.
>> 
>> 
>>> Probably we can do the same on the java side in this patch? it will simplify the native code a bit.
>> 
>> Yes, the code for mouseEvent looks similar to what we are trying to achieve for keyEvent but there are a few differences too-                 
>> - `mouseEvent(int lastX, int lastY, int buttonsState, boolean isButtonsDownState, boolean isMouseMove)`,  lastX  & lastY are already part of the parameters being sent from Java to native side.
>> 
>> - `keyEvent(int javaKeyCode, boolean keydown)` - Are you suggesting that a modifier flag state be sent from Java to native side? This would involve changing the present native `keyEvent` signature and adding code to native side to evaluate the new modifier flag state from Java and then map it to the corresponding [kCGEventFlagMask](https://developer.apple.com/documentation/coregraphics/cgeventflags/kcgeventflagmaskalphashift?language=objc).
>> 
>> Having everything on native side seems to be easier because with the present fix, the equivalent modifier flag mask is obtained using javaKeyCode + key state and a quick `NSDictionary modifierKeyToMaskMap` lookup without the need of adding or changing  keyEvent signature. 
>> 
>>> BTW do we need to set all this modifiers for the mouse clicks as well?
>> 
>> I don't think it is applicable for mouse clicks, as the issue (details below) occurs only when flagState is obtained using - `CGEventSourceFlagsState(kCGEventSourceStateHIDSystemState)` which was the case for keyEvents. I can double-check on this again.
>> 
>> **Issue:** 
>> _With the original code, the issue occurs at [CRobot.m#L295](https://github.com/openjdk/jdk/blob/ac6af6a64099c182e982a0a718bc1b780cef616e/src/java.desktop/macosx/native/libawt_lwawt/awt/CRobot.m#L295.) The flags gets reset or cleared when mouse is moved physically in unison with Robot's key events. The physical mouse movement causes the event flags to be reset._
>
>>I don't think it is applicable for mouse clicks, as the issue (details below) occurs only when flagState is obtained using - CGEventSourceFlagsState(kCGEventSourceStateHIDSystemState) which was the case for keyEvents. I can double-check on this again.
> 
> yes, please.
> if the physical mouse movement causes the "event flags" to be reset, then what about  "event flags" for mouse events

@mrserb @aivanov-jdk Based on previous suggestions, I have tested out few cases 

Following table list all the cases tested, along with expected & actual result in case of original and proposed solution.

- Case 1-5 are the ones already being tested as part of RobotModifierMaskTest.java
- Case 2 results in wrong output with original code (without mouse move).
- Case 6 is the mirrored case mentioned by @aivanov-jdk and I have verified that we are not setting or introducing new flag states here.
- Case 7-8 involve concurrent manual/ user + Robot interaction. This is bound to produce invalid or erroneous result due to simultaneous manual + Robot key events. With the proposed fix, the state of keyboard at time of initialization of Robot is taken into account and after initialization of Robot any modifier keys pressed by the user are ignored during Robot's activity. Hence we see 'A' for case 7 and 'a' for case 8-9 with proposed fix. I'm not completely sure if these cases are something we need to address since they fall under invalid scenarios for Robot testing/usage. @azuev-java can you please advice on cases 7-9.
 
So to answer previous question -

> There's also the fourth possibility: Robot generates keyboard events while the user presses modifier keys on the keyboard. It's an invalid scenario for tests but not impossible. Should the robot combine its internal state with the global hardware state?

With proposed fix we filter out any external (manual) modifier key events when Robot is typing and consider only the global hardware state only during initialization of Robot. 


| Case | Original Code (No External Mouse move) | With Proposed Fix (Without & With MouseMove) | Expected |
| --- | --- | --- | --- |
| 1. VK_SHIFT + VK_A - 3 times |  AAA |  AAA | AAA |
| 2. VK_CAPS + VK_A - 5 times <br> (caps lock toggled) | **aaaaa** | AaAaA| AaAaA| 
| 3. VK_META + VK_V - Copy clipboard content |  AAA |  AAA | AAA |
| 4. VK_ALT + VK_A - 3 times |  ååå | ååå | ååå |
| 5. VK_CONTROL + VK_A |  Caret position 0 |  Caret position 0 | Moves the caret to the start of the current line. |
| 6. Mirrored case -  Mouse moves triggered by Robot + concurrent external manual key events for above cases 1-5 |  Same result as above except case 2 - AaAaA | Produces same result as above. | Should work the same way as non-mirrored case - Robot key events + concurrent external manual mouse movements   |
|   |   |   |   |
| 7. VK_SHIFT is pressed & released at start of the application manually, before Robot types VK_A |  aaa | AAA | ??? |
| 8. VK_SHIFT is toggled manually after initialization of Robot, while Robot types VK_A - 5 times | AaAaA | aaaaa | ??? |
| 9. VK_CAPS is toggled manually after initialization of Robot, while Robot types VK_A - 5 times |  AaAaA | aaaaa | ??? |

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14744#issuecomment-1662976048