Multi-touch API

[Pavel Safrata]

Hello,

On 26.5.2012 10:36, Adam Granger wrote:
> Looks good, looking forward to this.
>
> I have some experience in this area... As part of some R&D at work I implemented multi-touch support in JavaFX and Swing using my own gesture processing with support for 1 or 2 finger swipe, pinch-to-zoom, rotate and hold.   I used the Win7 touch API via JNI to get raw events at the frame level, calculated which node (or component) was targeted and forwarded events to a "gesture recogniser". These were associated with a particular node using my own event manager. The "recognisers" allowed listeners with the classic begin, update, finish semantics, like those seen in Android. This is pretty much the MT4J architecture but open.
>
> I discovered during implementation that all the gesture processing could be widget-set agnostic (JavaFX vs Swing). There only ended up being a small plugin that was JavaFX/Swing specific allowing gesture-recognisers to be bound to nodes/components, and internally sort out the forwarding.
>
> Here's some code written from memory for javaFX
>
>    touchSupport = new JavaFxTouchSupport(jfxAppFrame);
>
>    touchSupport.addListener(fooNode, new PinchZoomRecogniser(new PinchZoomListener() {
>       public void beginZoom(event) {
>       }
>       public void updateZoom(event) {
>       }
>       public void finishZoom(event) {
>       }
>    }));
>    touchSupport.addListener(fooNode, new RotateRecogniser(new RotateListener() {
>       public void beginRotate(event) {
>       }
>       public void updateRotate(event) {
>       }
>       public void finishRotate(event) {
>       }
>    }));
>
> ...... and swing ....
>    touchSupport = new SwingTouchSupport(jFrame);
>
>    touchSupport.addListener(fooComponent, new SwipeRecogniser(new SwipeListener() {
>       public void beginSwipe(event) {
>       }
>       public void updateSwipe(event) {
>       }
>       public void finishSwipe(event) {
>       }
>    }));

I'll just state here that having API similar to Swing is not a goal of FX.

>
>
> Anyway, back to questions:
>
>   - Just wanted to confirm touch events will still be delivered even when the finger moves outside the node's bounds... and also the bounds of the JavaFX application? I found this necessary for pinch to zoom to "feel" right.

Yes, all the events are still delivered if the fingers are dragged 
outside of the node or even outside of the window.

>   - Is gesture recognition performed in JavaFX or does it just use whatever native mechanism is available - e.g. win7touch? I found it necessary to process in JavaFX as only the top level frame "exists" at the native platform level, i.e. nodes are just pixels, and frame level gestures were not useful as required two much translation.

We are using the native mechanisms as much as possible. On Windows we 
use the frame-level gestures. We discussed possible solutions of 
delivering the gestures on node basis (allowing for instance two nodes 
to be rotated simultaneously by two gestures), but it just doesn't fit 
to FX event system where the hierarchical events are bubbling through 
the hierarchical scenegraph. If you perform two rotations, it may mean 
dragging four leaf nodes, rotating two parents, zooming two entirely 
different nodes, any combination of it, we just don't know. Generating 
all the events would cause a great confusion. So we currently never 
generate two gestures of the same type simultaneously. Right now you can 
only use TouchEvents to recognize such node-based gestures (we tried it 
and it's not that big deal in fact). We plan for introducing a mechanism 
for registering custom gesture recognizers that will make it easier, but 
this doesn't apply to the nearest release.

>   - Is it possible to add your own gesture recognition for non-standard gestures, perhaps drawing a shape? - is it possible to capture touch events globally, in the same way you can in Swing/AWT using Toolkit.getDefaultToolkit().addAWTEventListener()?

As with all other events, you can register an event filter on scene and 
it will get all the events as soon as they are generated. As mentioned 
above, we plan for some nicer mechanism for registering custom gesture 
recognizers in the future.

>   - Multiple gestures on a single node... I had to a requirement to implement (two finger swipe + pinch-to-zoom) on a single node. This was tricky as I had to implement a locking mechanism where a particular touch point was "locked" into a gesture so couldn't be picked up by another accidentally. In JavaFX does either take precedence or are they processed in parallel or can it be customised? A real world example of this would be Maps on Android where pinch-to-zoom/rotate are both available on the map canvas... grab a phone, see what happens...

If you mean performing the two gestures by two fingers, there should be 
no such problem. The scroll, zoom and rotate gestures are generated 
simultaneously. If you touch two fingers and start moving, the node in 
the center gets the "started" event for all three gestures and then 
continues to receive all of them, regardless of where the fingers move.

If you mean performing the two gestures by four fingers, as mentioned 
above we don't have any special support for that so far.  If you decide 
to process the TouchEvents to recognize your own gestures, you can use 
the touch point ids to make sure you handle the correct ones.

>   - Is the code all JavaFX specific or is there any scope for back-porting to Swing using the architecture I described above?

The API is JavaFX specific.

>   - I never got onto to it, but is there any support/ideas for "lasso select"? i.e. draw a circle around a group of nodes...

I'm not sure what exactly do you mean. Do we have any "node selection" 
mechanism whatsoever? Do we even have a concept of a "selected" node 
(other than focus owner)? If you mean obtaining all nodes inside a curve 
drawn by a finger, we don't have any support for that, but again you can 
process the touches and compute whatever you want from them.

Thanks,
Pavel

>
> Regards,
>
> Adam.
>
>
>
>
> On 24 May 2012, at 08:48, Pavel Safrata wrote:
>
>> Hello,
>> we are at the end of a long journey to introduce an initial multi-touch support. The API has been lying in 2.2 repository for some time already, but we haven't got much feedback, so we've decided to make a last call for your concerns that might result in API changes before it goes public. We are already past feature freeze though, so the window is going to be quite small. A summary of the API follows.
>>
>> The touch actions produce three types of events: MouseEvents, GestureEvents and TouchEvents. All the events are delivered simultaneously and independently.
>>
>>
>> * Mouse events *
>>
>> Single touches are translated to normal MouseEvents, so the existing applications should work fine on touch screen. Sometimes you may need to identify and handle differently the mouse events synthesized from touch-screen, for that purpose they have a new isSynthesized flag.
>>
>>
>> * Gesture events *
>>
>> Gesture events are: ScrollEvent, RotateEvent, ZoomEvent, SwipeEvent. They are generated by both touch screen and trackpad. Basic common characteristics are:
>> * each event has coordinates, for trackpad events mouse coordinates are used, four touch screen events the center point between all the touches is used
>> * each event has modifiers information
>> * each event has isDirect flag that distinguishes between direct events produced by touching on direct coordinates on touch screen and indirect events produced by trackpad or mouse.
>>
>> The ScrollEvent, RotateEvent and ZoomEvent are continuous. They have event types "started", "performed" and "finished". The "started" event comes when the gesture is detected, then the "performed" events are delivered, containing delta values (change since the previous event) and total delta values (change since the gesture start). The "finished" event is delivered when the gesture finishes (the touches are released). After that, another "performed" events may be delivered, with an isInertia flag set to true. Whole gesture is delivered to a single node that was picked on gesture coordinates in time of gesture start.
>>
>> The SwipeEvent is a one-time event. When all the touch points involved in a gesture are pressed, then moved in the same direction and released, we recognize the gesture and deliver it as a single SwipeEvent (containing the swipe direction). Note that the described gesture produces also ScrollEvents, they are not exlusive with swipe.
>>
>> The gestures specifically:
>>
>> ScrollEvent has types SCROLL_STARTED, SCROLL, SCROLL_FINISHED. The started/finished notifications are generated only by touch gestures, mouse wheel still generates only one-time SCROLL event. In addition to the formerly existing deltaX and deltaY fields it has totalDeltaX and totalDeltaY (those contain zeros for mouse wheel scrolling). There is also a new field touchCount that specifies how many touch points are used for the gesture (a new gesture is started each time the touch count changes).
>>
>> ZoomEvent has types ZOOM_STARTED, ZOOM, ZOOM_FINISHED. It contains zoomFactor and totalZoomFactor. The values are to be multiplied with node's scale - values greater than 1 mean zooming in, values between zero and one mean zooming out.
>>
>> RotateEvent has types ROTATION_STARTED, ROTATE, ROTATION_FINISHED. It contains angle and totalAngle. The angles are in degrees and are meant to be added to node's rotation - positive values mean clock-wise rotation.
>>
>> SwipeEvent has types SWIPE_LEFT, SWIPE_RIGHT, SWIPE_UP, SWIPE_DOWN and contains touchCount field. On touch screen it is delivered to the node picked on center point of the entire gesture.
>>
>>
>> * Touch events *
>>
>> The TouchEvent can be used for tracking all the particular touch points. They are delivered for touch screen actions only, trackpad doesn't produce them.
>>
>> Each event carries a touch point (representing one pressed finger) and references to all other touch points. This design allows for handling and consuming each finger separately while making it possible to encapsulate handling of more complex multi-touch gestures in which not all touch points need to be over the handling node. In any moment of a multi-touch action, we have a set of touch points - for each of those touch points we create one touch event. This bunch of events is called "event set" and is marked by a common eventSetId number. All touch events from the set carry the same list of touch points, each of them carries different one as the "main" touch point.
>>
>> Each touch point has state (PRESSED, MOVED, STATIONARY, RELEASED), coordinates, id (unique in scope of a gesture) and target (the node to which this touch point's event is delivered). A method belongsTo(node) allows to test on the other touch points if they are delivered to the given target node (including bubbling).
>>
>> The event has types TOUCH_PRESSED, TOUCH_MOVED, TOUCH_STATIONARY, TOUCH_RELEASED, corresponding to the sate of its touch point. They also contain touchCount and modifiers.
>>
>> By default, each touch point is delivered to a single node during its whole trajectory, similarly to mouse dragging and gestures. This behavior is great for dragging nodes and is consistent with the rest of our events, but sometimes you want it different way - in this case it can be altered by using a grabbing API. The touch point provides methods ungrab() (since this call the touch point will be always delivered to a node picked under it), grab(node) (since this call the touch point will be always delivered to the given node) and grab() (since this call the touch point will be grabbed by the current source - the node whose handler calls it). In another words the grabbing/ungrabbing determines where to deliver each touch point and by default each newly pressed touch point is automatically grabbed by the node picked for it.
>>
>>
>> * Notes *
>>
>> There is one thing that makes the existing apps behave wrong. When you drag one finger over touch screen, it generates both mouse dragging and scrolling. We have to deliver both of them for the majority of applications to work correctly. The few nodes that handle both of those events (ScrollBar is a typical example) usually don't work well and need to be updated (to ignore synthesized mouse events for instance).
>>
>> For a classic application it should be enough to use mouse events and gestures. For a touch-screen-only appliction that wants to do some complex multi-touch logic, TouchEvents provide all the power. Developing an application that uses all kinds of events requires considerable care to avoid conflicting handling of user actions.
>>
>> We tried to stick with the native behavior as much as possible for users to get what they are used to. Where the underlying platform knows a gesture, we use the native recognition (elsewhere we have our own). We produce inertia for gestures based on the native support on each platform (for instance if a platform doesn't support zooming inertia at all, we don't generate it there). The goal is to make the same application behave as much native-like as possible on all platforms while minimizing developer's need to consider the differences.
>>
>> The gestures can generally be used without considering differences between touch-screen and trackpad. The most significant exception is touchCount on SwipeEvent. On touch-screen you can generate swipe by any number of touch points. On trackpad one-finger siwpes make no sense (they are used just to move the mouse cursor). On Mac in particular we use the native swipe recognition that generates only three and more finger swipes. So an application that uses touchCount on SwipeEvent will probably need different handling for direct and indirect events (which such complex applications will likely do anyway).
>>
>>
>> This is basically what we have in 2.2. Are there any objections?
>> Thanks,
>> Pavel