RFR: 8345538: Robot.mouseMove doesn't clamp bounds on macOS when trying to move mouse off screen [v4]

[Alisen Chung]

On Fri, 20 Dec 2024 19:36:20 GMT, Alisen Chung <achung at openjdk.org> wrote:

>> Currently on macOS when mouseMove is given an offscreen coordinate to move the mouse to, mouseMove will physically clamp to the edge of the screen, but if you try to grab the mouse location immediately after by using MouseInfo.getPointerInfo().getLocation() you will get the value of the offscreen point.
>> 
>> Windows and linux do this clamping and coordinate handling for us, but new distributions may not necessarily handle clamping the same way, so Robot should be checking for clamping rather than delegating it to native.
>> 
>> This fix updates shared code to cache the screen bounds and adds a check to not exceed the bounds in mouseMove. The caching is done in the Robot constructor, so if the screen bounds changes the constructor must be called again to update to the new bounds.
>
> Alisen Chung has updated the pull request incrementally with one additional commit since the last revision:
> 
>   update test

I did some further testing on macOS and found that as long as a second monitor is plugged in, if the mouse is moved off screen via robot.mouseMove (in either x or y coordinates) and the mouse position is queried via MouseInfo.getPointerInfo().getLocation(), getLocation will throw an NPE. 

I also tested windows and found different behavior from what Harshitha found - robot.mouseMove was already clamping on my primary screen and would no 

I was also able to reproduce the interesting corner case that Harshitha and Alexey described on macOS:
Primary screen on right, secondary on left -> move mouse to (200, 20000)
macOS - mouse moves to right of primary screen, NPE
I don’t have any scaling on my macOS so I’m not sure how a rounding error could occur in my setup.

I discussed the multi-monitor issue with Phil offline and I am working on an implementation that should work on multi-monitor setups. The plan is to store each of the screenBounds rectangles instead of just the main screen, then on mouseMove, loop through each of the screens until a coordinate lands within the bounds of a screen.  

The question is when the coordinate lands out of bounds of all screens, especially in cases where it would be reasonable to clamp to either screen, what should the behavior be? We discussed a few options:
1) we could look for the closest screen in the x direction, then y direction and clamp on that screen using x and y
2) we could look for the closest screen in both directions by calculating absolute distance then clamp on that screen using x and y
3) we could look for the closest screen with absolute distance then calculate the slope to figure out where to stop (basically “fly” there and once you hit a wall you stop)

Currently on macOS the mouse tries to “fly” over to the location and moves along the first wall it encounters, so a combination of 2 and 3. The problem I could see with this approach is that it relies not only on the final location but also the starting location, so a mouseMove to the same location would have different behaviors depending on where you started. I’m not sure if this is a problem but @aivanov-jdk and @honkar-jdk do you have any thoughts/opinions?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/22781#issuecomment-2605522152