RFR: JDK-8302618: Problem typing uppercase letters with java.awt.Robot on Mac when moving mouse
Harshitha Onkar
honkar at openjdk.org
Fri Jun 30 22:39:05 UTC 2023
**Problem:**
Robot erroneously produces lowercase letter when mouse is moved (manually) in unison with Robot's keyEvents on MacOS. This issue was originally logged by a developer of an on-screen accessibility keyboard - TouchBoard. Originally reported at https://github.com/adoptium/adoptium-support/issues/710
The issue is reproducible on JDK versions 22 to 11, but works fine on JDK-8.
This issue is not restricted for the Shift modifier key and causes problems with other modifier keys too and sometimes without any external mouse movement.
- This works correctly on JDK-8 up to JDK-9+129 when Accessibility APIs (AXUIElementCreateSystemWide/ AXUIElementPostKeyboardEvent) were used. Later on it was changed to CGEvents.
- CGEvent APIs were introduced in [jdk-9+130](https://github.com/openjdk/jdk/commit/0d7becfcb7568417afb9c19f2db24e7797afc1d6#diff-ad38f25bd9f72b45c5cca2ba7f7fc6743c4820e7aeb1bbf5d7802c9d4e0e1479).
- With the present code, the issue occurs at [CRobot.m#L295](https://github.com/openjdk/jdk/blob/ac6af6a64099c182e982a0a718bc1b780cef616e/src/java.desktop/macosx/native/libawt_lwawt/awt/CRobot.m#L295.) The flags gets reset or cleared when mouse is moved physically in unison with Robot's key events.
- The physical mouse movement causes the event flags to be reset.
**Impact:**
Modifier keys don't work as expected when using Robot with any simultaneous physical mouse movement and in case of TouchBoard, this behavior breaks the usability of the on-screen a11y keyboard. There is no known workaround for this particular use case except for reverting to JDK-8. More details on this use case [here.](https://github.com/adoptium/adoptium-support/issues/710#issuecomment-1594103280)
**Solution**
- In order to avoid the resetting of the CGEventFlags here [CRobot.m#L295](https://github.com/openjdk/jdk/blob/ac6af6a64099c182e982a0a718bc1b780cef616e/src/java.desktop/macosx/native/libawt_lwawt/awt/CRobot.m#L295.), the CGEvent flag state is obtained in `initRobot` (stored in initFlags) which is later used within `CRobot_keyEvent`.
- The incoming keyCode is used to determine whether it is a modifier key and the corresponding modifierFlagMask is either added or cleared from the initFlags based on whether the modifier key was pressed or released.
- Finally, only the required and known flag bits from initFlag are copied over to local flag which is used in `CGEventSetFlags()`.
**Testing**
The newly added test - RobotModifierMaskTest.java tests for Shift, Caps, Control, Option and Command keys.
It should be tested for two cases:
CASE 1 : Run the test as an automated test WITHOUT any mouse movement and let the Robot go through all the test cases.
CASE 2 : Run the test in semi-automated mode. While the Robot in typing, manually move the mouse (without clicking/dragging). check if the test Passes or Fails.
CI Testing works as expected with the fix and does not break any existing Robot related tests.
-------------
Commit messages:
- whitespace fix
- test update
- removed debug logs
- test changes & allModifiersMask chnaged to static var
- copyright header for test
- localFlag fix, allModifiersMask added for correct flag bit calculation
- initial test case added with debug logs
- spacing
- Dictionary implementation changes with NSLog
- indent changes
- ... and 1 more: https://git.openjdk.org/jdk/compare/ec45bd64...20f8fd07
Changes: https://git.openjdk.org/jdk/pull/14744/files
Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=14744&range=00
Issue: https://bugs.openjdk.org/browse/JDK-8302618
Stats: 288 lines in 4 files changed: 284 ins; 0 del; 4 mod
Patch: https://git.openjdk.org/jdk/pull/14744.diff
Fetch: git fetch https://git.openjdk.org/jdk.git pull/14744/head:pull/14744
PR: https://git.openjdk.org/jdk/pull/14744
More information about the client-libs-dev
mailing list