Touch Events, Scrolling, and Windows Pen

[Danno Ferrin]

On Tue, May 28, 2013 at 2:44 AM, Anthony Petrov
<anthony.petrov at oracle.com>wrote:

> Hi Danno,
>
>
> On 05/27/2013 07:27 PM, Danno Ferrin wrote:
>
>> My next problem is the pen device on Windows.  The only events captured in
>> JavaFX from a pen are mouse events.  To complicate things the
>> "Synthesized"
>> flag is set.  This is a bit disconcerting because to the view of a JFX app
>> there is no other event for it to be synthesized to.  So I would propose
>> the isPenEvent method in /glass/glass-lib-windows/src/**ViewContainer.cpp
>> be
>> added to check that the 0x80 bit is set by changing the signature
>> to 0xFF515780 and the mask to 0xFFFFFF80, making a pen event look just
>> like
>> a mouse event, since it only generates mouse events. (the linked MSDN
>> article in the code explains the lower 8 bits, and the high bit is a
>> pen/touch flag, the logic may need to be more complex).  To distinguish a
>> pen tap perhaps we set the direct flag but not the synthesized flag, since
>> a pen is direct on the screen (usually).
>>
>> So am I way off base on these?  Should I work up a patch for these?  Or
>> must the current behavior be maintained.
>>
>
> AFAIK, we haven't tested Glass/FX with a Pen device and I doubt we even
> have proper hardware to test it at the moment. So such a patch would
> certainly be welcome. Please feel free to post it on this mailing list for
> a review.
>

I think the QA and dev team should invest a few thousand and buy one or two
each of a Surface Pro, a Samsung Ativ 500T, and maybe an Asus VivoTab (but
only the Asus if you can get the pen with it on the same order).  The
"triple threat" input devices (mouse/pen/touch) will cover a lot of bases.


> However, we should first decide how we want to process Pen events. They
> are not regular mouse events, nor are they similar to regular touch events
> (though they have a lot in common with the latter). E.g. suppose you can
> control your Surface Pro with a finger using its touch screen. These are
> touch events. Now, you could also connect a USB mouse to this device, and
> that would have to generate regular mouse events. Additionally, you could
> connect a graphic tablet such as a Wacom Bamboo for example, and this
> device would have to generate some special Pen events, right?
>

As far as I can tell pen evens are just the same as mouse events unless you
use microsofts "Ink" APIs to insert "Ink" into your application.  So unless
JavaFX adds support for digital ink, there is vey little difference. The
only one that comes to mind is pressure sensitivity, and the minority of
windows pen screens or pads support pressure sensitivity.



>
> Should we introduce a new kind of events for Pen events in FX? Or does it
> make sense to use touch events for this purpose? How would we distinguish
> between finger- and pen- based input events in the above scenario then?
>

Behavior is a big difference in pen vs touch.  Tap and drag for touch
gets interpreted as a scroll pan, while tap and drag for a pen
gets interpreted as a mouse click and drag, for example selecting text.

Pen events don't generate touch gestures like the touch screens do.  Unless
you count the "flicks" api in microsoft land which just generates events at
the windows level for stuff like delete, cut, copy, paste, etc.  Much like
the multimedia keyboard.

A new boolean field could be added isPen could be added on the mouseEvent,
or a new enum property at InputEvent called Input Device, that would
enumerate if it was mouse, touch, pen, keyboard, or whatever standard input
devies show up in the future.  These enums could then be loaded with all
sorts of interesting queries in case a user wants to write their app in a
future proof way, such as are drag events principally pan-scrolling, is the
gesture device direct/indirect.  Maybe not an enum, but a separate HCI
object rather than pushing the random flags into the event system.


> What other options do we have?
>
>
Keeping pen input second class is fine I think, as long as it is consistent
with the differences between touch and mouse in non-ink areas.  If we have
to confuse them with one type or the other, confuse them with mouse events
because they are closer to mouse events then touch events. There are more
important areas for JavaFX to go to before it goes down the digital ink
path.


> --
> best regards,
> Anthony
>