<AWT Dev> Proposal on enhancing MouseWheelEvent to include absolute scrolling delta

[Pavel Fatin]

Hi Sergey,

The absolute deltas are /optional /and provided only when they are 
available, so there's no need to generate these values on all platforms, 
for all events. For example, mouse wheel doesn't generate absolute 
deltas, because a wheel step naturally corresponds to a line (or several 
discrete lines). We can use 0.0 value to indicate the absence of 
absolute delta (so, if some platform doesn't provide absolute deltas, 
0.0 is a valid value, and the platform is still "supported").

The Apple's scrollingDeltaY description tells 
<https://developer.apple.com/reference/appkit/nsevent/1535387-scrollingdeltay>: 
"When hasPreciseScrollingDeltas is false, /multiply /the value returned 
by this method by the line or row height. Otherwise /scroll by /the 
returned amount". That particular API uses a /single /variable to hold 
both relative- and absolute deltas, together with the additional flag 
that tells how to interpret the value (i.e. when 
hasPreciseScrollingDeltas is false, scrollingDeltaY contains deltaY). 
It's just a slightly different way to represent the two kinds of data, 
using a separate boolean flag, instead of the 0.0 marker (i.e. it puts 
the two disparate values into a single variable and then provides a way 
to distinguish them).

Apple doesn't explicitly specify units for the absolute deltas (it only 
tells to "scroll /by/ the returned amount"). The Windows' Direct 
Manipulation API states that the absolute transformations are given in 
device-independent pixels. It seems that we can also specify 
"device-independent pixels" explicitly.

There's no reliable way to convert absolute deltas to relative deltas, 
because the relative deltas are /relative /to arbitrary scrolling unit 
size, which is not a part of input, but a part of the content rendering 
(which might not even exist). OS generates the two kinds of data 
separately, and both spatial- and temporal resolutions differ 
considerably (so, an attempt to somehow channel the absolute delta into 
the already existing, relative one, can potentially raise compatibility 
/ performance problems).

We have /already /implemented "smooth scrolling" in IDEA itself. One 
doesn't even need high-precision input to render the transitions 
smoothly, yet high-resolution input is useful to increase the 
positioning precision, to transfer the dynamics better and to reduce the 
input latency.

Besides, there's no much sense in trying to "squeeze" the absolute delta 
into the relative one as "a temporary solution" – just as well, we can 
temporary add the absolute deltas to JetBrains Runtime directly, which 
seems to be a more reliable and a better-performing solution that really 
delivers Mac-like scrolling feel.

The central purpose of the proposal is not to create some temporary 
workaround for a particular task, but rather to enhance and improve the 
Java itself, so it can be up-to-date with the state of modern hardware / 
OSes. If JDK 9 is already frozen, I hope that the idea can still be 
useful for the next versions.


On 11.01.2017 16:18, Sergey Bylokhov wrote:
> H, Pavel.
> Thanks for this proposal.
> First of all if you have a plan to implement smooth scrolling in the 
> jdk9, then it will be good to use only existing API. (We already had a 
> the feature freeze).
> So for now a solution will be better to implement the fix via 
> getPreciseWheelRotation 
> <https://docs.oracle.com/javase/8/docs/api/java/awt/event/MouseWheelEvent.html#getPreciseWheelRotation-->(). 
> At least in this case you will be able to implement it only on 
> Mac/win, but if you provide the new API then it should be supported on 
> all platforms.
> Is it possible to convert absolute data to the deltas? What is the 
> difference between absolute data and deltas? The main problem which I 
> see here is that we have no strict definition of units which we use 
> for scrolling(ticks, pixels, lines or 2-pixels on the retina etc).
>
>> Hi All,
>>
>> Next follows a proposal on enhancingMouseWheelEventto include 
>> absolute scrolling delta.
>>
>> My name is Pavel Fatin, I work at JetBrains (IntelliJ IDEA). Although 
>> this is my first message here, I have been working with Java / AWT / 
>> Swing since Java 1.2 and have some in-depth experience (for example, 
>> my research onlow-latency painting in AWT and Swing 
>> <https://pavelfatin.com/low-latency-painting-in-awt-and-swing/>).
>>
>> Because we at JetBrains receive manyrequests 
>> <https://youtrack.jetbrains.com/issue/IDEA-76396>to implement 
>> smooth-scrolling in IDEA, as an experiment, weextended 
>> <https://github.com/JetBrains/intellij-community/commit/34b9dfd0585937c3731e06a89554d1dc86f7f235>our 
>> custom scroll pane class to 
>> handleMouseWheelEvent'sgetPreciseWheelRotation() 
>> <https://docs.oracle.com/javase/8/docs/api/java/awt/event/MouseWheelEvent.html#getPreciseWheelRotation-->data 
>> (that method was introduced in Java 7, but it’s not yet used by AWT / 
>> Swing directly). As a result, the scrolling "smoothness" and 
>> precision improved substantially, however on OS X it was still a far 
>> cry from the native scrolling experience, particularly:
>>
>>  *
>>
>>     Scrolling speed was unit-dependent (while in OS X it's not). The
>>     concept of "wheel rotation" is inherently tied to the idea of
>>     "scrolling unit" (usually 1 or 3 lines). This makes sense for a
>>     real mouse scrolling wheel, but doesn't make much sense for a
>>     touchpad – one expects a uniform mapping between physical touch
>>     gesture and a screen update, consistent across different
>>     applications. The same goes for touchscreen interfaces.
>>     Introduction of the fractional wheel rotations increases
>>     precision but still retains the "rotation-unit" bound.
>>
>>  *
>>
>>     Spatial precision was lacking. Typical scrolling deltas were in
>>     range 0.5..>10, and this was only a slight improvement over
>>     1..>10 deltas.
>>
>>  *
>>
>>     Temporal resolution was lacking, so the scrolling was rather
>>     coarse-grained and there was a substantial lag between starting a
>>     gesture and a reaction on the screen. This mostly stems from the
>>     insufficient spatial resolution. Added interpolation helped to
>>     artificially increase spatial- and temporal resolutions, but
>>     couldn't solve the input lag and the lack of positioning precision.
>>
>> Wondering how the native OS X applications can perform much better we 
>> started to investigate the Cocoa API. It turns out, that since Mac OS 
>> X 10.7 (Lion)NSEvent 
>> <https://developer.apple.com/reference/appkit/nsevent>, in addition 
>> to thedeltaX 
>> <https://developer.apple.com/reference/appkit/nsevent/1534871-deltax>anddeltaY 
>> <https://developer.apple.com/reference/appkit/nsevent/1534158-deltay>introducedscrollingDeltaX 
>> <https://developer.apple.com/reference/appkit/nsevent/1524505-scrollingdeltax>andscrollingDeltaY 
>> <https://developer.apple.com/reference/appkit/nsevent/1535387-scrollingdeltay>properties 
>> that supply high-resolution scrolling information 
>> (withhasPrecisionScrollingDeltas 
>> <https://developer.apple.com/reference/appkit/nsevent/1525758-hasprecisescrollingdeltas>property 
>> that specifies how to interpret the data). So far so good, but those 
>> properties are quite a different beast – they 
>> contain/absolute/scrolling deltas rather than relative deltas, 
>> supplied by the previous values.
>>
>
> Can you please clarify this. Its is unclear from the specification 
> what is the difference between deltaX, 
> scrollingDeltaX(hasPrecisionScrollingDeltas=true), 
> scrollingDeltaX(hasPrecisionScrollingDeltas=false).
>
>> How can we integrate such values in the existing Java API?
>>
>> One way is to try to somehow translate the absolute deltas to 
>> relative "wheel rotations", but the wheel event per se represents 
>> input data and doesn't deal with possible scrolling units as such. 
>> Another problem with this approach is that OS might provide so-called 
>> "pixel-perfect" scrolling, where smallest delta is guaranteed to be 1 
>> pixel (yet still preserving large-distance scrolling via OS-level 
>> acceleration), as OS knows real physical resolution of particular 
>> input / output devices and the acceleration curve, and such a 
>> translation cannot reliably handle this. And one more problem with 
>> the masking is compatibility – because the new data have much more 
>> temporal resolution, existing applications that process precise wheel 
>> events synchronously might be overwhelmed, hog CPU and lag 
>> unnecessarily. All in all, "partial wheel rotation" is an imperfect 
>> and "leaky" abstraction for the absolute scrolling delta.
>>
>> A possible "hack" is to put the absolute values directly into the 
>> relative delta property and then to hardcode unit size to be 1 on Mac 
>> OS (that is what GTK+ 3does 
>> <https://github.com/GNOME/gtk/blob/master/gtk/gtkscrolledwindow.c#L1245>), 
>> but that is what it is – a hack, not compatible with the existing 
>> codebase.
>>
>> It seems, that the most reasonable solution is to introduce a new, 
>> "absolute" kind of scrolling deltas, in addition to the relative one. 
>> That is, for example, whatQWheelEvent <https://goo.gl/kZqxVc>does 
>> with itsangleDelta <https://goo.gl/zwxBt9>andpixelDelta 
>> <https://goo.gl/F5A37V>properties. We tried that approach in our 
>> JetBrains Runtime byextending 
>> <https://github.com/JetBrains/jdk8u_jdk/commit/568f2dae82b0fe27b79ce6943071d89463758610>MouseWheelEventwithgetScrollingDelta()method 
>> andimplementing 
>> <https://github.com/JetBrains/jdk8u_jdk/commit/a3cb8807b148879e9c70a74a8a16c30a28991581>it 
>> in Mac OS – the result is a genuine, OSX-like scrolling experience in 
>> Java.
>>
>> We chose the method name “getScrollingDelta” (a la Apple) instead of 
>> “getPixelDelta” (a la Qt) because the input event by itself doesn't 
>> necessarily results in display scrolling, and because the concept of 
>> "pixel" is not that precise. Likewise, we chosedoubletype for further 
>> extensibility, considering the case withgetWheelDelta()(NSEventalso 
>> uses a floating point value).
>>
>> It should be noted, that the absolute delta/complements/rather than 
>> replaces the relative value. Those two values are not interchangeable 
>> and OS generates the values separately (for legacy devices only the 
>> relative delta is generated and, in such cases, we can use 0 as an 
>> absolute one). Addition of the absolute scrolling delta fully 
>> preserves the functioning of Java's existing event processing mechanisms.
>>
>> Although currently only Mac OS supplies that kind of data, Windows 
>> Precision Touchpad and XInput2 / libinput might follow, as smooth 
>> scrolling becomes a thing nowadays. Additionally, touchscreen-based 
>> scrolling needs precisely this kind of scrolling deltas.
>>
>> While we may implement this functionality privately in our JetBrains 
>> Runtime, we believe that more people can benefit from it. Moreover, 
>> the enhancement ofMouseWheelEventto support pixel-perfect scrolling 
>> can pave a road for smooth scrolling in AWT and Swing (which is a 
>> good thing to improve usability, both GTK+ 3 and Qt 5 already support 
>> that).
>>
>> -- 
>> Sincerely, Pavel
>

-- 
Sincerely, Pavel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/awt-dev/attachments/20170112/df1b1871/attachment-0001.html>