<AWT Dev> Proposal on enhancing MouseWheelEvent to include absolute scrolling delta

[Pavel Fatin]

Proposal on enhancing MouseWheelEvent to include absolute scrolling delta

Hi All,

Next follows a proposal on enhancing MouseWheelEvent to include absolute 
scrolling delta.

My name is Pavel Fatin, I work at JetBrains (IntelliJ IDEA). Although 
this is my first message here, I have been working with Java / AWT / 
Swing since Java 1.2 and have some in-depth experience (for example, my 
research on low-latency painting in AWT and Swing 
<https://pavelfatin.com/low-latency-painting-in-awt-and-swing/>).

Because we at JetBrains receive many requests 
<https://youtrack.jetbrains.com/issue/IDEA-76396> to implement 
smooth-scrolling in IDEA, as an experiment, we extended 
<https://github.com/JetBrains/intellij-community/commit/34b9dfd0585937c3731e06a89554d1dc86f7f235> 
our custom scroll pane class to handle MouseWheelEvent's 
getPreciseWheelRotation() 
<https://docs.oracle.com/javase/8/docs/api/java/awt/event/MouseWheelEvent.html#getPreciseWheelRotation--> 
data (that method was introduced in Java 7, but it’s not yet used by AWT 
/ Swing directly). As a result, the scrolling "smoothness" and precision 
improved substantially, however on OS X it was still a far cry from the 
native scrolling experience, particularly:

  *

    Scrolling speed was unit-dependent (while in OS X it's not). The
    concept of "wheel rotation" is inherently tied to the idea of
    "scrolling unit" (usually 1 or 3 lines). This makes sense for a real
    mouse scrolling wheel, but doesn't make much sense for a touchpad –
    one expects a uniform mapping between physical touch gesture and a
    screen update, consistent across different applications. The same
    goes for touchscreen interfaces. Introduction of the fractional
    wheel rotations increases precision but still retains the
    "rotation-unit" bound.

  *

    Spatial precision was lacking. Typical scrolling deltas were in
    range 0.5..>10, and this was only a slight improvement over 1..>10
    deltas.

  *

    Temporal resolution was lacking, so the scrolling was rather
    coarse-grained and there was a substantial lag between starting a
    gesture and a reaction on the screen. This mostly stems from the
    insufficient spatial resolution. Added interpolation helped to
    artificially increase spatial- and temporal resolutions, but
    couldn't solve the input lag and the lack of positioning precision.

Wondering how the native OS X applications can perform much better we 
started to investigate the Cocoa API. It turns out, that since Mac OS X 
10.7 (Lion) NSEvent 
<https://developer.apple.com/reference/appkit/nsevent>, in addition to 
the deltaX 
<https://developer.apple.com/reference/appkit/nsevent/1534871-deltax> 
and deltaY 
<https://developer.apple.com/reference/appkit/nsevent/1534158-deltay> 
introduced scrollingDeltaX 
<https://developer.apple.com/reference/appkit/nsevent/1524505-scrollingdeltax> 
and scrollingDeltaY 
<https://developer.apple.com/reference/appkit/nsevent/1535387-scrollingdeltay> 
properties that supply high-resolution scrolling information (with 
hasPrecisionScrollingDeltas 
<https://developer.apple.com/reference/appkit/nsevent/1525758-hasprecisescrollingdeltas> 
property that specifies how to interpret the data). So far so good, but 
those properties are quite a different beast – they contain /absolute/ 
scrolling deltas rather than relative deltas, supplied by the previous 
values.

How can we integrate such values in the existing Java API?

One way is to try to somehow translate the absolute deltas to relative 
"wheel rotations", but the wheel event per se represents input data and 
doesn't deal with possible scrolling units as such. Another problem with 
this approach is that OS might provide so-called "pixel-perfect" 
scrolling, where smallest delta is guaranteed to be 1 pixel (yet still 
preserving large-distance scrolling via OS-level acceleration), as OS 
knows real physical resolution of particular input / output devices and 
the acceleration curve, and such a translation cannot reliably handle 
this. And one more problem with the masking is compatibility – because 
the new data have much more temporal resolution, existing applications 
that process precise wheel events synchronously might be overwhelmed, 
hog CPU and lag unnecessarily. All in all, "partial wheel rotation" is 
an imperfect and "leaky" abstraction for the absolute scrolling delta.

A possible "hack" is to put the absolute values directly into the 
relative delta property and then to hardcode unit size to be 1 on Mac OS 
(that is what GTK+ 3 does 
<https://github.com/GNOME/gtk/blob/master/gtk/gtkscrolledwindow.c#L1245>), 
but that is what it is – a hack, not compatible with the existing codebase.

It seems, that the most reasonable solution is to introduce a new, 
"absolute" kind of scrolling deltas, in addition to the relative one. 
That is, for example, what QWheelEvent <https://goo.gl/kZqxVc> does with 
its angleDelta <https://goo.gl/zwxBt9> and pixelDelta 
<https://goo.gl/F5A37V> properties. We tried that approach in our 
JetBrains Runtime by extending 
<https://github.com/JetBrains/jdk8u_jdk/commit/568f2dae82b0fe27b79ce6943071d89463758610> 
MouseWheelEvent with getScrollingDelta() method and implementing 
<https://github.com/JetBrains/jdk8u_jdk/commit/a3cb8807b148879e9c70a74a8a16c30a28991581> 
it in Mac OS – the result is a genuine, OSX-like scrolling experience in 
Java.

We chose the method name “getScrollingDelta” (a la Apple) instead of 
“getPixelDelta” (a la Qt) because the input event by itself doesn't 
necessarily results in display scrolling, and because the concept of 
"pixel" is not that precise. Likewise, we chose double type for further 
extensibility, considering the case with getWheelDelta() (NSEvent also 
uses a floating point value).

It should be noted, that the absolute delta /complements/ rather than 
replaces the relative value. Those two values are not interchangeable 
and OS generates the values separately (for legacy devices only the 
relative delta is generated and, in such cases, we can use 0 as an 
absolute one). Addition of the absolute scrolling delta fully preserves 
the functioning of Java's existing event processing mechanisms.

Although currently only Mac OS supplies that kind of data, Windows 
Precision Touchpad and XInput2 / libinput might follow, as smooth 
scrolling becomes a thing nowadays. Additionally, touchscreen-based 
scrolling needs precisely this kind of scrolling deltas.

While we may implement this functionality privately in our JetBrains 
Runtime, we believe that more people can benefit from it. Moreover, the 
enhancement of MouseWheelEvent to support pixel-perfect scrolling can 
pave a road for smooth scrolling in AWT and Swing (which is a good thing 
to improve usability, both GTK+ 3 and Qt 5 already support that).

-- 
Sincerely, Pavel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/awt-dev/attachments/20170104/c2ec8c65/attachment.html>