Beyond the Screen:

The Future of User Interaction in a Wearable World

Erica Stanley ( @ericastanley )

Screens, Touch/ Pointer

@ericastanley

#BeyondScreens

Look Ma, No Screens

@ericastanley

#BeyondScreens

Interaction Models

@ericastanley

#BeyondScreens

Title Text

OUI Interaction Model

@ericastanley

#BeyondScreens

Title Text

OUI Interaction Model

@ericastanley

#BeyondScreens

Developing Voice Interfaces

• One of the oldest means of human interaction
  • Human-computer interaction imitating very natural human-human communication
• Requires no hand movement or body effort

Benefits

@ericastanley

#BeyondScreens

Underlying Components

@ericastanley

#BeyondScreens

Underlying Components

Initiates AudioStream,

captured by device

@ericastanley

#BeyondScreens

Underlying Components

Initiates AudioStream,

captured by device

Calls on experiences shared

by device and user.

@ericastanley

#BeyondScreens

The vision for your system will determine how you put these components together.

Developing Voice Interfaces

• A set of vocabulary of commands 
• focus is on recognition component

Bounded, structured experience

@ericastanley

#BeyondScreens

Developing Voice Interfaces

Bounded, structured experience

• Benefits

  • fast, would be able to perform recognition quickly 

  • limited --> less space required

  • very accurate

  • consistent

@ericastanley

#BeyondScreens

Developing Voice Interfaces

Bounded, structured experience

• Disadvantages

  • system is inflexible

  • user bears responsibility for knowing commands

  • designer's job to bring commands to user's attention at appropriate time

@ericastanley

#BeyondScreens

Developing Voice Interfaces

Richer Dialogue

• System takes initiative

• Can think of these interfaces more as a human-human conversation

• How are the user's supported in this dialogue?

  • Awareness of task, immediate context, memory of very recent behavior

• Learns users preferences over time and the dialogue grows accordingly

@ericastanley

#BeyondScreens

Developing Voice Interfaces

Best Practices

• When to use:

  • Any use case where touch interaction is not desired and user's visual focus is low

    • In-car interaction and navigation

      • Eyes and hands are occupied

    • Medical environments requiring sterility

  •  Security

    • Powerful mechanism for identifying user

  • Shared experiences

@ericastanley

#BeyondScreens

Developing Voice Interfaces

Best Practices

• When to avoid:

  • Public Places

  • Interactions where errors can have life-threatening impact

@ericastanley

#BeyondScreens

Title Text

OUI Interaction Model

@ericastanley

#BeyondScreens

Developing Manipulation-based

Interfaces

• Natural real-world intuitive interactions (NUI)
• 2 Main types
  • Deformable
  • Non-deformable
    • screen based touch
    • texting
    • tangible handling

@ericastanley

#BeyondScreens

Developing Manipulation-based

Interfaces

• Touch Type:  tip, pad, nail, knuckle
• Number of simultaneous touch points
• Motion of the touch
• Direction of motion
• Amount of pressure used
• Duration
• Distance
• Location of contact points

Touch, Non-deformable (NUI)

@ericastanley

#BeyondScreens

Developing Manipulation-based

Interfaces

• Input interactions that users perform using freehand manipulations that change the shape of a deformable object or display
• Rely on skills users already have and use in real world activities

Deformable (OUI)

@ericastanley

#BeyondScreens

@ericastanley

#BeyondScreens

Developing Manipulation-based

Interfaces

• When to Use:
  • Physical form compliments/ enhances function
  • Precision/ accuracy is desired
  • Public places where voice or gesture tracking would be difficcult

Best Practices

@ericastanley

#BeyondScreens

@ericastanley

#BeyondScreens

Developing Manipulation-based

Interfaces

• When to Avoid:
  • Hands are likely occupied
    • Driving/ Cycling
  • Situations requiring sterility
    • Medical situations
    • Serving/ Eating food

Best Practices

@ericastanley

#BeyondScreens

@ericastanley

#BeyondScreens

Developing Manipulation-based

Interfaces

• In 2D spaces, ensure at least a 15mm surface radius for interactive elements and at least 5mm between
• In 3D spaces, consider visual feedback (ex. depth) for movable elements
• Consider haptic feedback when visual feedback isn't available/ desired
• Respond to every contact with immediate feedback and fluid transitions
• Enable basic, direct manipulation
  • Avoid secondary controls, when possible

Best Practices

@ericastanley

#BeyondScreens

@ericastanley

#BeyondScreens

Title Text

OUI Interaction Model

@ericastanley

#BeyondScreens

Developing Gesture-based

Interfaces

• In-Air Gestures (OUI)
• Like Speech, one of the oldest forms of human-human communication
• Used to issue a command or communicate  with devices
• Must design system to capture gestures
  • Hand movements
  • Facial Expressions
  • Body Motion
  • Eye Tracking
• Makes use of Computer Vision and Machine Learning

@ericastanley

#BeyondScreens

Developing Gesture-based

Interfaces

• When to Use
  • Interaction with large displays
  • Situations requiring sterility
• When to Avoid
  • Public places

  • Interactions where errors can have life-threatening impact

@ericastanley

#BeyondScreens

Developing Gesture-based

Interfaces

• Best Practices
  • Avoid time-based gestures as it delays power users
  • Clearly communicate how to start and stop a gesture
  • Apply the gesture principles for a learnable safe system 
    • identity
    • negation
    • inversion
    • commutativity
  • Handle false gestures, positive & negative

@ericastanley

#BeyondScreens

Augmented Reality/ Mixed Reality Interfaces

• Overlaying user interface to enhance the body or physical environment
  • Visual Augmented Interfaces
    • Projected Display in conjunction with camera display(s) of physical environment
    • Makes use of Computer Vision to track gestures and targets in the user's environment
    • May also enhance user's field of vision and focus
  • Audio Augmented Interfaces
    • Overlaying audio on top of environmental sounds
    • May also enhance user's audible or vocal range
  • Mixed modality (combination interfaces)

@ericastanley

#BeyondScreens

Interaction Patterns

@ericastanley

#BeyondScreens

Microinteractions

Contained product moments that revolve around a single use case and have one main task.

@ericastanley

#BeyondScreens

M*C Based Patterns

• MCRpd
• OUI Framework
• Model Presentation Adapter Controller Coordinator

@ericastanley

#BeyondScreens

MCRpd

@ericastanley

#BeyondScreens

OUI Framework

@ericastanley

#BeyondScreens

MPACC: Model-Presentation-Adapter-Controller-Coordinator

@ericastanley

#BeyondScreens

OUI Interaction Patterns

• Microinteractions simplify interaction with non-display devices
• M*C patterns reduce coupling between models, controllers and interfaces
  • Reusability - enables development for multiple connected devices/ platform rather than a single device
  • Separation of responsibility - simplifies development process
• MPACC
  • Best for Connected System Architecture
• OUI Framework (updated MCRpd) 
  • ​Best for Connected Device Architecture

Considerations

@ericastanley

#BeyondScreens

Sources

1. http://airccse.org/journal/ijasuc/papers/4413ijasuc04.pdf
2. http://www.oreilly.com/pub/e/3146
3. http://goo.gl/7NDlBb
4. http://www.oreilly.com/pub/e/3223
5. http://gaia.cs.uiuc.edu/papers/ubicomp01-c.pdf
6. http://goo.gl/YmIrCj
7. http://praxtime.com/2014/03/31/god-particle-augmented-audio-reality-wearables/