Understanding Flutter's

handling of Gestures

Guillaume DIALLO

@Guitoof

@guitoof

Head of

Flutter Engineering

1

Handling Classic Gestures

2

Handling
Custom Gestures

3

Overlapping Gesture Handlers

2.2

Writing a custom GestureRecognizer

2.1

The Gesture
Detection Flow

4

Testing Gestures

3.1

HitTest
behaviors

3.2

Gesture Disambiguation

Classic Gestures Handling

# GestureDetector

Gesture

 

Detector

GestureDetector

Higher level API for gesture detection

GestureDetector

Gesture Detector

Tap

# GestureDetector

Double Tap

Horizontal Drag

Long Press

Vertical Drag

Tap

Long Press

Horizontal Drag

Vertical Drag

Double Tap

Scale

Force Press

# GestureDetector

Pan

Handling Custom Gestures

# RawGestureDetector

Raw

Detector

GestureDetector

RawGestureDetector

RawGestureDetector

Detect gestures from custom gesture recognizers

Gesture

 

GestureDetector

GestureRecognizer

<<interface>>

RawGestureDetector

...

# Custom GestureRecognizer

TapGestureRecognizer

DoubleTapGestureRecognizer

LongPressGestureRecognizer

HorizontalDragGestureRecognizer

VerticalDragGestureRecognizer

PanGestureRecognizer

ScaleGestureRecognizer

GestureDetector

GestureRecognizer

<<interface>>

TapGestureRecognizer

RawGestureDetector

...

CustomGestureRecognizer

# Custom GestureRecognizer

DoubleTapGestureRecognizer

LongPressGestureRecognizer

HorizontalDragGestureRecognizer

VerticalDragGestureRecognizer

PanGestureRecognizer

ScaleGestureRecognizer

Gesture detection/recognition

flow

Pointers & Events

Gesture events life cycle

# Gesture Detection Flow

  • PointerDownEvent

  • PointerMoveEvent

  • PointerUpEvent

  • PointerRemovedEvent

  • PointerAddedEvent

Pointer Events

+ hitTest

GestureBinding

+ handlePointerEvent

1

PlatformDispatcher

Entry point for

platform related events

1

Propagate Hit Test

# Gesture Detection Flow

Hit Testing

Hit testing down the Render Tree

hitTest ?

hitTest ?

hitTest ?

HitTestResult = 

# Gesture Detection Flow

Pointer Events

+ hitTest

handleEvent()

handleEvent()

RenderPointerListener

GestureBinding

+ handlePointerEvent

1

2

Call handleEvent on each target

Listener

RawGestureDetector

3

<Any>GestureRecognizer

+ addPointer

_handlePointerDown

Call addPointer on each recognizer

Register route 

PointerRouter

+ addRoute

+ dispatchEvent

+ handleEvent

Call all registered routes

<_handleAnyPointerEvent>

Actually handle event

4

RendererBinding

Propagate hitTest through the render tree

+ hitTest

PlatformDispatcher

Entry point for

platform related events

1

Propagate Hit Test

2

Dispatch Event

3

Register pointer events routes

4

Call registered routes to handles the event from recognizers

+ handleEvent

<<HitTestTarget>>

# Gesture Detection Flow

Custom Gesture Recognizer

# Custom GestureRecognizer

Custom GestureRecognizer

V-Shape Gesture Recognizer

V


class VShapeGestureRecognizer extends GestureRecognizer { 
}

Custom GestureRecognizer

Implement

# Custom GestureRecognizer
# Custom GestureRecognizer

Custom GestureRecognizer

Implement


class VShapeGestureRecognizer extends GestureRecognizer {

  @override
  void acceptGesture(int pointer) {}
  @override
  void rejectGesture(int pointer) {}
  
}
			

class VShapeGestureRecognizer extends GestureRecognizer {
  @override 
  void addAllowedPointer(PointerEvent event) {
    GestureBinding.instance.pointerRouter.addRoute(
      event.pointer, _handleEvent, event.transform
    );
  }
  
  void _handleEvent(PointerEvent event) {
    if(event is PointerDownEvent) {
      /// handle pointer down event
    } else if (...) {
      /// or handle any other pointer event as you need here
    }
  }
  
  ...

  @override
  void acceptGesture(int pointer) {}
  @override
  void rejectGesture(int pointer) {}
  
}
			
# Custom GestureRecognizer

Custom GestureRecognizer

Register route to handle pointer events


class VShapeGestureRecognizer extends GestureRecognizer {
  final _tracker = ShapeTracker<VShape>();

  @override 
  void addAllowedPointer(PointerEvent event) {
    GestureBinding.instance.pointerRouter.addRoute(
      event.pointer, _handleEvent, event.transform
    );
  }
  
  void _handleEvent(PointerEvent event) {
    if(event is PointerDownEvent || event is PointerMoveEvent) {
      _tracker.track(event);
    } else if (...) {
      /// or handle any other pointer event as you need here
    }
  }
  
  ...

  @override
  void acceptGesture(int pointer) {}
  @override
  void rejectGesture(int pointer) {}
  
}
			
# Custom GestureRecognizer

Custom GestureRecognizer

Add some V-shape tracker


class VShapeGestureRecognizer extends GestureRecognizer {

  final _tracker = ShapeTracker<VShape>();

  void Function(VShapeDetails details)? onVShapeDetected;

  @override 
  void addAllowedPointer(PointerEvent event) {
    GestureBinding.instance.pointerRouter.addRoute(
      event.pointer, _handleEvent, event.transform
    );
  }
  
  void _handleEvent(PointerEvent event) {
    if(event is PointerDownEvent || event is PointerMoveEvent) {
      _tracker.track(event);
    } else if(event is PointerUpEvent && _tracker.hasValidShape) {
      onVShapeDetected?.call(_tracker.shapeDetails);
    }
  }
  
  ...

  @override
  void acceptGesture(int pointer) {}
  @override
  void rejectGesture(int pointer) {}
  @override
  String get debugDescription () => 'Custom Gesture';
  
}
			
# Custom GestureRecognizer

Custom GestureRecognizer

Trigger callback if tracked shape is valid on pointer Up event


...
return RawGestureDetector(
  gestures: {
    VShapeGestureRecognizer: GestureRecognizerFactoryWithHandlers<VShapeGestureRecognizer>(
      () => VShapeGestureRecognizer(),
      (VShapeGestureRecognizer instance) {
        instance.onVShapeDetected = ( VShapeDetails details) {
          /// Do stuff to handle the detected V-shape here...
    	}
      },
    ),
  }
);
			
# Custom GestureRecognizer

Custom GestureRecognizer

Use with a RawGestureDetector

# Custom GestureRecognizer

Custom GestureRecognizer

V-Shape Gesture Recognizer

Overlapping

Gesture handlers

Button
Triggers some action in the app

Transparent Gesture
Handler layer

Trigger some Analytics event

Button
❌ Triggers some action in the app

Transparent Gesture
Handler layer

✅ Trigger some Analytics event

# Overlapping Gesture Handlers

What happens with multiple targets ?

Hit Testing

Hit Test Behavior

1. Should I pass the hit test through to my children?

2. Should I add myself to the list of widgets that were hit?

3. Should I tell my parent that I considered the hit test to hit me or my children?

deferToChild

translucent

opaque

# Hit Test Behavior

Hit Testing

Actual: Hit Test Behavior

HitTestResult = 

RenderObject 1

RenderObject 2

deferToChild

transluscent

opaque

deferToChild

transluscent

opaque

# Hit Test Behavior

Hit Testing

Hit Test Behavior: example

deferToChild

transluscent

opaque

Stack

A

B

Circle

Stack

A

B

Circle (opaque)

HitTestBehavior for B

# Hit Test Behavior

Hit Testing

Hit Test Behavior: example

deferToChild

transluscent

opaque

HitTestBehavior

for B

# Hit Test Behavior

Who actually handles the event?

OR

Disambiguation

Gesture Arena

A

B

HitTestResult =

GestureArena

class AnyGestureRecognizer extends GestureRecognizer {
  ...
  GestureBinding.instance.gestureArena.add(pointer, this);
  ...
}
			
class AnyGestureRecognizer extends GestureRecognizer {
  ...
  gestureArenaEntry.resolve(GestureDisposition.accepted|rejected);
  ...
}

1 winner

OR

# Gesture Arena

Disambiguation

Gesture Arena

# Gesture Arena

Craig Labenz
Gesture Arena | Decoding Flutter

How to handle gesture from multiple recognizers ?

AND

# Listener

Listener

GestureDetector

RawGestureDetector

Listener

Lowest gestures handling API

deals with pointer events

Listener

GestureDetector

GestureRecognizer

<<interface>>

RawGestureDetector

...

# Custom GestureRecognizer

TapGestureRecognizer

DoubleTapGestureRecognizer

LongPressGestureRecognizer

HorizontalDragGestureRecognizer

VerticalDragGestureRecognizer

PanGestureRecognizer

ScaleGestureRecognizer

Listener

GestureDetector

Listener

# Listener

Listener

Listener vs GestureDetector ?

Listener

Handle raw pointer events


return Listener(
  child: child,
  onPointerDown: (PointerDownEvent event) => {}
  onPointerMove: (PointerMoveEvent event) => {}
  onPointerUp: (PointerUpEvent event) => {}
  onPointerSignal: (PointerSignalEvent event) => {}
  onPointerHover: (PointerHoverEvent event) => {}
  onPointerCancel: (PointerCancelEvent event) => {}
  behavior: HitTestBehavior.deferToChild|translucent|opaque,
);

			
# Listener

return Scaffold(,
  floatingActionButton: FloatingActionButton(
    child: const Icon(...),
    onPressed: () {...}
  ),
);
        
# Listener

Listener

Detect gestures events without disturbance


return Listener(
  child: Scaffold(,
    floatingActionButton: FloatingActionButton(
      child: const Icon(...),
      onPressed: () {...}
    ),
  ),
);
        
# Listener

Listener

Detect gestures events without disturbance

return Listener(
  /// Handle any pointer related events from the Listener...
  onPointerDown: (PointerDownEvent event) => {}
  onPointerMove: (PointerMoveEvent event) => {}
  onPointerUp: (PointerUpEvent event) => {}
  child: Scaffold(,
    floatingActionButton: FloatingActionButton(
      child: const Icon(...),
      /// ...and handle any regular gesture normally
      onPressed: () {...}
    ),
  ),
);
        
# Listener

Listener

Detect gestures events without disturbance

# Listener

Listener

Detect gestures events without disturbance

How to Test

Gestures?

# Gesture Testing

WidgetTester

WidgetController

Simulate high level gestures in a testing environment 

Testing

High level WidgetTester API : classic gestures

testWidgets(
  'should call "onTap" when receiving a tap gesture',
  (WidgetTester tester) async {
    final tapCallback = MockGestureCallback();
    await tester.pumpWidget(GestureDetector(
      onTap: tapCallback,
    ));
    
    await tester.tap(find.byType(GestureDetector));

    verify(() =>tapCallback()).called(1);
  },
);
			
tap(Finder finder);
longPress(Finder finder);
drag(Finder finder, Offset offset);
fling(Finder finder, Offset offset,
double speed);

Emulate gestures on Widgets

Emulate gestures at/from locations

tapAt(Offset position);
longPressAt(Offset position);
dragFrom(Offset position, Offset offset);
flingFrom(Offset position, Offset offset,
double speed);
			
# Gesture Testing
# Gesture Testing

WidgetTester

WidgetController

Simulate high level gestures in a testing environment 

TestGesture

+ createGesture

Higher level test gesture API

Lower level test gesture API

...

+ down

+ up

+ moveBy

+ moveTo

+ tap

+ longPress

+ drag

+ fling

+ tapAt

+ longPressAt

+ dragFrom

+ flingFrom

testWidgets(
  'should call "onVerticalDrag..." callbacks when receiving a drag gesture oriented vertically',
  (WidgetTester tester) async {
    final gestureCallback = MockGestureCallback();
    await tester.pumpWidget(GestureDetector(
      onVerticalDragStart: gestureCallback,
      onVerticalDragUpdate: gestureCallback,
      onVerticalDragEnd: gestureCallback,
    ));

    const startPosition = Offset(100, 100);    
    final gesture = await tester.startGesture(startPosition);
    gesture.moveBy(const  Offset(0, 100));
    gesture.up();

    verify(() =>gestureCallback(any())).called(3);
  },
);
			
# Gesture Testing

Testing

Low level WidgetTester API : custom gestures

TestWidgetsFlutterBinding

Test gestures can be recognized by any detector that needs to be tested

WidgetTester

WidgetController

Simulate high level gestures in a testing environment 

GestureBinding

+ handlePointerEvent

+ sendEventToBinding

....

TestGesture

+ createGesture

Higher level test gesture API

Lower level test gesture API

All test gestures end up in the GestureBinding's handlePointerEvent

...

+ down

+ up

+ moveBy

+ moveTo

GestureDetector

RawGestureDetector

Listener

# Gesture Testing

+ tap

+ longPress

+ drag

+ fling

+ tapAt

+ longPressAt

+ dragFrom

+ flingFrom

Wrap up

GestureDetector

GestureRecognizer

<<interface>>

RawGestureDetector

CustomGestureRecognizer

Listener

Detect most usual gestures

Detect & Recognize any custom gestures

Handle multiple gestures simultaneously without interfering

Test usual & custom gestures

with a high & low level gesture testing API

WidgetTester

# Wrap up

Useful links

  • https://github.com/
    guitoof/gestures_handling_sandbox
  • @Hixie's answer on Issue #74733
    HitTestBehavior.opaque documentation is wrong
  • GestureBinding & Gesture Disambiguation documentation
    https://docs.flutter.dev/ui/advanced/gestures#gesture-disambiguation
    https://api.flutter.dev/flutter/gestures/GestureBinding-mixin.html
  • Craig's Decoding Flutter
    video on Gesture Arena

Thank you

@Guitoof

@guitoof

Questions ?

Made with Slides.com