<AWT Dev> Proposal on enhancing MouseWheelEvent to include absolute scrolling delta

Wed Jan 11 15:18:54 UTC 2017

H, Pavel.
Thanks for this proposal. 
First of all if you have a plan to implement smooth scrolling in the jdk9, then it will be good to use only existing API. (We already had a the feature freeze).
So for now a solution will be better to implement the fix via getPreciseWheelRotation <https://docs.oracle.com/javase/8/docs/api/java/awt/event/MouseWheelEvent.html#getPreciseWheelRotation-->(). At least in this case you will be able to implement it only on Mac/win, but if you provide the new API then it should be supported on all platforms.
Is it possible to convert absolute data to the deltas? What is the difference between absolute data and deltas? The main problem which I see here is that we have no strict definition of units which we use for scrolling(ticks, pixels, lines or 2-pixels on the retina etc).

> Hi All,
> Next follows a proposal on enhancing MouseWheelEvent to include absolute scrolling delta.
> My name is Pavel Fatin, I work at JetBrains (IntelliJ IDEA). Although this is my first message here, I have been working with Java / AWT / Swing since Java 1.2 and have some in-depth experience (for example, my research on low-latency painting in AWT and Swing <https://pavelfatin.com/low-latency-painting-in-awt-and-swing/>).
> Because we at JetBrains receive many requests <https://youtrack.jetbrains.com/issue/IDEA-76396> to implement smooth-scrolling in IDEA, as an experiment, we extended <https://github.com/JetBrains/intellij-community/commit/34b9dfd0585937c3731e06a89554d1dc86f7f235> our custom scroll pane class to handle MouseWheelEvent's getPreciseWheelRotation() <https://docs.oracle.com/javase/8/docs/api/java/awt/event/MouseWheelEvent.html#getPreciseWheelRotation--> data (that method was introduced in Java 7, but it’s not yet used by AWT / Swing directly). As a result, the scrolling "smoothness" and precision improved substantially, however on OS X it was still a far cry from the native scrolling experience, particularly:
> Scrolling speed was unit-dependent (while in OS X it's not). The concept of "wheel rotation" is inherently tied to the idea of "scrolling unit" (usually 1 or 3 lines). This makes sense for a real mouse scrolling wheel, but doesn't make much sense for a touchpad – one expects a uniform mapping between physical touch gesture and a screen update, consistent across different applications. The same goes for touchscreen interfaces. Introduction of the fractional wheel rotations increases precision but still retains the "rotation-unit" bound.
> Spatial precision was lacking. Typical scrolling deltas were in range 0.5..>10, and this was only a slight improvement over 1..>10 deltas.
> Temporal resolution was lacking, so the scrolling was rather coarse-grained and there was a substantial lag between starting a gesture and a reaction on the screen. This mostly stems from the insufficient spatial resolution. Added interpolation helped to artificially increase spatial- and temporal resolutions, but couldn't solve the input lag and the lack of positioning precision.
> Wondering how the native OS X applications can perform much better we started to investigate the Cocoa API. It turns out, that since Mac OS X 10.7 (Lion) NSEvent <https://developer.apple.com/reference/appkit/nsevent>, in addition to the deltaX <https://developer.apple.com/reference/appkit/nsevent/1534871-deltax> and deltaY <https://developer.apple.com/reference/appkit/nsevent/1534158-deltay> introduced scrollingDeltaX <https://developer.apple.com/reference/appkit/nsevent/1524505-scrollingdeltax> and scrollingDeltaY <https://developer.apple.com/reference/appkit/nsevent/1535387-scrollingdeltay>properties that supply high-resolution scrolling information (with hasPrecisionScrollingDeltas <https://developer.apple.com/reference/appkit/nsevent/1525758-hasprecisescrollingdeltas> property that specifies how to interpret the data). So far so good, but those properties are quite a different beast – they contain absolute scrolling deltas rather than relative deltas, supplied by the previous values.

Can you please clarify this. Its is unclear from the specification what is the difference between deltaX, scrollingDeltaX(hasPrecisionScrollingDeltas=true), scrollingDeltaX(hasPrecisionScrollingDeltas=false).

> How can we integrate such values in the existing Java API?
> One way is to try to somehow translate the absolute deltas to relative "wheel rotations", but the wheel event per se represents input data and doesn't deal with possible scrolling units as such. Another problem with this approach is that OS might provide so-called "pixel-perfect" scrolling, where smallest delta is guaranteed to be 1 pixel (yet still preserving large-distance scrolling via OS-level acceleration), as OS knows real physical resolution of particular input / output devices and the acceleration curve, and such a translation cannot reliably handle this. And one more problem with the masking is compatibility – because the new data have much more temporal resolution, existing applications that process precise wheel events synchronously might be overwhelmed, hog CPU and lag unnecessarily. All in all, "partial wheel rotation" is an imperfect and "leaky" abstraction for the absolute scrolling delta.
> A possible "hack" is to put the absolute values directly into the relative delta property and then to hardcode unit size to be 1 on Mac OS (that is what GTK+ 3 does <https://github.com/GNOME/gtk/blob/master/gtk/gtkscrolledwindow.c#L1245>), but that is what it is – a hack, not compatible with the existing codebase.
> It seems, that the most reasonable solution is to introduce a new, "absolute" kind of scrolling deltas, in addition to the relative one. That is, for example, what QWheelEvent <https://goo.gl/kZqxVc> does with its angleDelta <https://goo.gl/zwxBt9> and pixelDelta <https://goo.gl/F5A37V> properties. We tried that approach in our JetBrains Runtime by extending <https://github.com/JetBrains/jdk8u_jdk/commit/568f2dae82b0fe27b79ce6943071d89463758610> MouseWheelEvent with getScrollingDelta() method and implementing <https://github.com/JetBrains/jdk8u_jdk/commit/a3cb8807b148879e9c70a74a8a16c30a28991581> it in Mac OS – the result is a genuine, OSX-like scrolling experience in Java.
> We chose the method name “getScrollingDelta” (a la Apple) instead of “getPixelDelta” (a la Qt) because the input event by itself doesn't necessarily results in display scrolling, and because the concept of "pixel" is not that precise. Likewise, we chose doubletype for further extensibility, considering the case with getWheelDelta() (NSEvent also uses a floating point value).
> It should be noted, that the absolute delta complements rather than replaces the relative value. Those two values are not interchangeable and OS generates the values separately (for legacy devices only the relative delta is generated and, in such cases, we can use 0 as an absolute one). Addition of the absolute scrolling delta fully preserves the functioning of Java's existing event processing mechanisms.
> Although currently only Mac OS supplies that kind of data, Windows Precision Touchpad and XInput2 / libinput might follow, as smooth scrolling becomes a thing nowadays. Additionally, touchscreen-based scrolling needs precisely this kind of scrolling deltas.
> While we may implement this functionality privately in our JetBrains Runtime, we believe that more people can benefit from it. Moreover, the enhancement of MouseWheelEvent to support pixel-perfect scrolling can pave a road for smooth scrolling in AWT and Swing (which is a good thing to improve usability, both GTK+ 3 and Qt 5 already support that).
> -- 
> Sincerely, Pavel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/awt-dev/attachments/20170111/f7335112/attachment-0001.html>