Touch Events, Scrolling, and Windows Pen

[Danno Ferrin]

I created a new bug for the pen support bug, with a tested patch.  I spent
way more time getting the build up than doing the fix, such is native work.
https://javafx-jira.kenai.com/browse/RT-30869

For the pan-select, I attached a fix for list views to the existing bug.
If that approach passes muster I can do the same for tree and table.
https://javafx-jira.kenai.com/browse/RT-21091

On Tue, May 28, 2013 at 10:46 AM, Anthony Petrov
<anthony.petrov at oracle.com>wrote:

> Sounds good to me.
>
> Danno, could you please prepare a patch for this bug, test it, and post it
> for review here?
>
> --
> best regards,
> Anthony
>
>
> On 05/28/2013 07:46 PM, Danno Ferrin wrote:
>
>> For 8.0 that would be the best solution IMHO.
>> On May 28, 2013 8:41 AM, "Pavel Safrata" <pavel.safrata at oracle.com>
>> wrote:
>>
>>  So it seems to me that in short term (FX8) the pen should produce mouse
>>> events with isSynthesized==false. We can introduce something more
>>> advanced
>>> in the future, but right now it should generally work and the synthesized
>>> flag really seems to be misused here.
>>> Pavel
>>>
>>> On 28.5.2013 16:09, Danno Ferrin wrote:
>>>
>>>  On Tue, May 28, 2013 at 2:44 AM, Anthony Petrov
>>>> <anthony.petrov at oracle.com>****wrote:
>>>>
>>>>   Hi Danno,
>>>>
>>>>>
>>>>>
>>>>> On 05/27/2013 07:27 PM, Danno Ferrin wrote:
>>>>>
>>>>>   My next problem is the pen device on Windows.  The only events
>>>>> captured
>>>>>
>>>>>> in
>>>>>> JavaFX from a pen are mouse events.  To complicate things the
>>>>>> "Synthesized"
>>>>>> flag is set.  This is a bit disconcerting because to the view of a JFX
>>>>>> app
>>>>>> there is no other event for it to be synthesized to.  So I would
>>>>>> propose
>>>>>> the isPenEvent method in /glass/glass-lib-windows/src/*****
>>>>>>
>>>>>> *ViewContainer.cpp
>>>>>> be
>>>>>> added to check that the 0x80 bit is set by changing the signature
>>>>>> to 0xFF515780 and the mask to 0xFFFFFF80, making a pen event look just
>>>>>> like
>>>>>> a mouse event, since it only generates mouse events. (the linked MSDN
>>>>>> article in the code explains the lower 8 bits, and the high bit is a
>>>>>> pen/touch flag, the logic may need to be more complex).  To
>>>>>> distinguish
>>>>>> a
>>>>>> pen tap perhaps we set the direct flag but not the synthesized flag,
>>>>>> since
>>>>>> a pen is direct on the screen (usually).
>>>>>>
>>>>>> So am I way off base on these?  Should I work up a patch for these?
>>>>>>  Or
>>>>>> must the current behavior be maintained.
>>>>>>
>>>>>>   AFAIK, we haven't tested Glass/FX with a Pen device and I doubt we
>>>>>> even
>>>>>>
>>>>> have proper hardware to test it at the moment. So such a patch would
>>>>> certainly be welcome. Please feel free to post it on this mailing list
>>>>> for
>>>>> a review.
>>>>>
>>>>>   I think the QA and dev team should invest a few thousand and buy one
>>>>> or
>>>>>
>>>> two
>>>> each of a Surface Pro, a Samsung Ativ 500T, and maybe an Asus VivoTab
>>>> (but
>>>> only the Asus if you can get the pen with it on the same order).  The
>>>> "triple threat" input devices (mouse/pen/touch) will cover a lot of
>>>> bases.
>>>>
>>>>
>>>>   However, we should first decide how we want to process Pen events.
>>>> They
>>>>
>>>>> are not regular mouse events, nor are they similar to regular touch
>>>>> events
>>>>> (though they have a lot in common with the latter). E.g. suppose you
>>>>> can
>>>>> control your Surface Pro with a finger using its touch screen. These
>>>>> are
>>>>> touch events. Now, you could also connect a USB mouse to this device,
>>>>> and
>>>>> that would have to generate regular mouse events. Additionally, you
>>>>> could
>>>>> connect a graphic tablet such as a Wacom Bamboo for example, and this
>>>>> device would have to generate some special Pen events, right?
>>>>>
>>>>>   As far as I can tell pen evens are just the same as mouse events
>>>>> unless
>>>>>
>>>> you
>>>> use microsofts "Ink" APIs to insert "Ink" into your application.  So
>>>> unless
>>>> JavaFX adds support for digital ink, there is vey little difference. The
>>>> only one that comes to mind is pressure sensitivity, and the minority of
>>>> windows pen screens or pads support pressure sensitivity.
>>>>
>>>>
>>>>
>>>>   Should we introduce a new kind of events for Pen events in FX? Or
>>>> does it
>>>>
>>>>> make sense to use touch events for this purpose? How would we
>>>>> distinguish
>>>>> between finger- and pen- based input events in the above scenario then?
>>>>>
>>>>>   Behavior is a big difference in pen vs touch.  Tap and drag for touch
>>>>>
>>>> gets interpreted as a scroll pan, while tap and drag for a pen
>>>> gets interpreted as a mouse click and drag, for example selecting text.
>>>>
>>>> Pen events don't generate touch gestures like the touch screens do.
>>>>   Unless
>>>> you count the "flicks" api in microsoft land which just generates events
>>>> at
>>>> the windows level for stuff like delete, cut, copy, paste, etc.  Much
>>>> like
>>>> the multimedia keyboard.
>>>>
>>>> A new boolean field could be added isPen could be added on the
>>>> mouseEvent,
>>>> or a new enum property at InputEvent called Input Device, that would
>>>> enumerate if it was mouse, touch, pen, keyboard, or whatever standard
>>>> input
>>>> devies show up in the future.  These enums could then be loaded with all
>>>> sorts of interesting queries in case a user wants to write their app in
>>>> a
>>>> future proof way, such as are drag events principally pan-scrolling, is
>>>> the
>>>> gesture device direct/indirect.  Maybe not an enum, but a separate HCI
>>>> object rather than pushing the random flags into the event system.
>>>>
>>>>
>>>>   What other options do we have?
>>>>
>>>>>
>>>>>
>>>>>   Keeping pen input second class is fine I think, as long as it is
>>>>>
>>>> consistent
>>>> with the differences between touch and mouse in non-ink areas.  If we
>>>> have
>>>> to confuse them with one type or the other, confuse them with mouse
>>>> events
>>>> because they are closer to mouse events then touch events. There are
>>>> more
>>>> important areas for JavaFX to go to before it goes down the digital ink
>>>> path.
>>>>
>>>>
>>>>   --
>>>>
>>>>> best regards,
>>>>> Anthony
>>>>>
>>>>>
>>>>>
>>>