Beyond the Screen:

The Future of User Interaction in a Wearable World

Erica Stanley ( @ericastanley )

Screens, Touch/ Pointer

@ericastanley

#BeyondScreens

Look Ma, No Screens

@ericastanley

#BeyondScreens

Interaction Models

@ericastanley

#BeyondScreens

Title Text

OUI Interaction Model

@ericastanley

#BeyondScreens

Title Text

OUI Interaction Model

@ericastanley

#BeyondScreens

Developing Voice Interfaces

  • One of the oldest means of human interaction
    • Human-computer interaction imitating very natural human-human communication
  • Requires no hand movement or body effort

Benefits

@ericastanley

#BeyondScreens

Underlying Components

@ericastanley

#BeyondScreens

Underlying Components

Initiates AudioStream,

captured by device

@ericastanley

#BeyondScreens

Underlying Components

Initiates AudioStream,

captured by device

Calls on experiences shared

by device and user.

@ericastanley

#BeyondScreens

The vision for your system will determine how you put these components together.

Developing Voice Interfaces

  • A set of vocabulary of commands 
  • focus is on recognition component

Bounded, structured experience

@ericastanley

#BeyondScreens

Developing Voice Interfaces

Bounded, structured experience

  • Benefits

    • fast, would be able to perform recognition quickly 

    • limited --> less space required

    • very accurate

    • consistent

@ericastanley

#BeyondScreens

Developing Voice Interfaces

Bounded, structured experience

  • Disadvantages

    • system is inflexible

    • user bears responsibility for knowing commands

    • designer's job to bring commands to user's attention at appropriate time

@ericastanley

#BeyondScreens

Developing Voice Interfaces

Richer Dialogue

  • System takes initiative

  • Can think of these interfaces more as a human-human conversation

  • How are the user's supported in this dialogue?

    • Awareness of task, immediate context, memory of very recent behavior

  • Learns users preferences over time and the dialogue grows accordingly

@ericastanley

#BeyondScreens

Developing Voice Interfaces

Best Practices

  • When to use:

    • Any use case where touch interaction is not desired and user's visual focus is low

      • In-car interaction and navigation

        • Eyes and hands are occupied

      • Medical environments requiring sterility

    •  Security

      • Powerful mechanism for identifying user

    • Shared experiences

@ericastanley

#BeyondScreens

Developing Voice Interfaces

Best Practices

  • When to avoid:

    • Public Places

    • Interactions where errors can have life-threatening impact

@ericastanley

#BeyondScreens

Title Text

OUI Interaction Model

@ericastanley

#BeyondScreens

Developing Manipulation-based

Interfaces

  • Natural real-world intuitive interactions (NUI)
  • 2 Main types
    • Deformable
    • Non-deformable
      • screen based touch
      • texting
      • tangible handling

@ericastanley

#BeyondScreens

Developing Manipulation-based

Interfaces

  • Touch Type:  tip, pad, nail, knuckle
  • Number of simultaneous touch points
  • Motion of the touch
  • Direction of motion
  • Amount of pressure used
  • Duration
  • Distance
  • Location of contact points

 

Touch, Non-deformable (NUI)

@ericastanley

#BeyondScreens

Developing Manipulation-based

Interfaces

  • Input interactions that users perform using freehand manipulations that change the shape of a deformable object or display
  • Rely on skills users already have and use in real world activities

Deformable (OUI)

@ericastanley

#BeyondScreens

@ericastanley

#BeyondScreens

Developing Manipulation-based

Interfaces

  • When to Use:
    • Physical form compliments/ enhances function
    • Precision/ accuracy is desired
    • Public places where voice or gesture tracking would be difficcult

Best Practices

@ericastanley

#BeyondScreens

@ericastanley

#BeyondScreens

Developing Manipulation-based

Interfaces

  • When to Avoid:
    • Hands are likely occupied
      • Driving/ Cycling
    • Situations requiring sterility
      • Medical situations
      • Serving/ Eating food

Best Practices

@ericastanley

#BeyondScreens

@ericastanley

#BeyondScreens

Developing Manipulation-based

Interfaces

  • In 2D spaces, ensure at least a 15mm surface radius for interactive elements and at least 5mm between
  • In 3D spaces, consider visual feedback (ex. depth) for movable elements
  • Consider haptic feedback when visual feedback isn't available/ desired
  • Respond to every contact with immediate feedback and fluid transitions
  • Enable basic, direct manipulation
    • Avoid secondary controls, when possible

Best Practices

@ericastanley

#BeyondScreens

@ericastanley

#BeyondScreens

Title Text

OUI Interaction Model

@ericastanley

#BeyondScreens

Developing Gesture-based

Interfaces

  • In-Air Gestures (OUI)
  • Like Speech, one of the oldest forms of human-human communication
  • Used to issue a command or communicate  with devices
  • Must design system to capture gestures
    • Hand movements
    • Facial Expressions
    • Body Motion
    • Eye Tracking
  • Makes use of Computer Vision and Machine Learning

@ericastanley

#BeyondScreens

Developing Gesture-based

Interfaces

  • When to Use
    • Interaction with large displays
    • Situations requiring sterility
  • When to Avoid
    • Public places

    • Interactions where errors can have life-threatening impact

@ericastanley

#BeyondScreens

Developing Gesture-based

Interfaces

  • Best Practices
    • Avoid time-based gestures as it delays power users
    • Clearly communicate how to start and stop a gesture
    • Apply the gesture principles for a learnable safe system 
      • identity
      • negation
      • inversion
      • commutativity
    • Handle false gestures, positive & negative

@ericastanley

#BeyondScreens

Augmented Reality/ Mixed Reality Interfaces

  • Overlaying user interface to enhance the body or physical environment
    • Visual Augmented Interfaces
      • Projected Display in conjunction with camera display(s) of physical environment
      • Makes use of Computer Vision to track gestures and targets in the user's environment
      • May also enhance user's field of vision and focus
    • Audio Augmented Interfaces
      • Overlaying audio on top of environmental sounds
      • May also enhance user's audible or vocal range
    • Mixed modality (combination interfaces)

@ericastanley

#BeyondScreens

Interaction Patterns

@ericastanley

#BeyondScreens

Microinteractions

Contained product moments that revolve around a single use case and have one main task.

@ericastanley

#BeyondScreens

M*C Based Patterns

  • MCRpd
  • OUI Framework
  • Model Presentation Adapter Controller Coordinator

@ericastanley

#BeyondScreens

MCRpd

@ericastanley

#BeyondScreens

OUI Framework

@ericastanley

#BeyondScreens

MPACC: Model-Presentation-Adapter-Controller-Coordinator

@ericastanley

#BeyondScreens

OUI Interaction Patterns

  • Microinteractions simplify interaction with non-display devices
  • M*C patterns reduce coupling between models, controllers and interfaces
    • Reusability - enables development for multiple connected devices/ platform rather than a single device
    • Separation of responsibility - simplifies development process
  • MPACC
    • Best for Connected System Architecture
  • OUI Framework (updated MCRpd) 
    • ​Best for Connected Device Architecture

Considerations

@ericastanley

#BeyondScreens

Sources

  1. http://airccse.org/journal/ijasuc/papers/4413ijasuc04.pdf
  2. http://www.oreilly.com/pub/e/3146
  3. http://goo.gl/7NDlBb
  4. http://www.oreilly.com/pub/e/3223
  5. http://gaia.cs.uiuc.edu/papers/ubicomp01-c.pdf
  6. http://goo.gl/YmIrCj
  7. http://praxtime.com/2014/03/31/god-particle-augmented-audio-reality-wearables/
Made with Slides.com