Natural User Interfaces

What NUI is?
eXperimenting with NUI

Touch devices

We know it well, mobile and tablet devices..

Types of NUIs

Voice or Speech recognition

Like Google Voice API for Google Search, YouTube etc.

Kinect Voice recognition

Types of NUIs

Touch free or Gestures Recognition Systems

Nintendo Wii, PlayStation Move

Leap Motion

Kinect

Types of NUIs

Other systems

Gaze tracking (Eye movement tracking)

Brain Machine Interfaces (BCI)

Types of NUIs

Why do we like NUIs?

They provide a multi-sensory experience. It’s our gestures, body movements and voice
They enhance user’s control over the interface
Users don’t have to be computer expert in order to handle the interface. It is supposed to be designed for natural interaction!
They support simultaneous interaction by more than one user
They can be fun and entertaining and they are encourage exploration

Where can they go wrong?

There aren’t any established conventions. Apart from some design patterns in multi-touch devices, other interfaces don’t have a set of guidelines yet. This can cause some confusion among users
There’s lack of feedback. For example, there isn’t always any indicator of where the finger is pointing towards. GUIs have the mouse cursor.
It’s very difficult for complex systems to be used solely with gesture-based interfaces.
Same gestures mean different things in different cultures. Therefore, there’s a need for cross-cultural conventions
For all the above reasons, users might be easily annoyed, bored or tired if they can’t figure out how to efficiently control the interface

More about Gesture recognition -

How is it working?

Kinect use infrared sensor to generate depth data, which is useful to recognize objects like human bodys.

More about Gesture recognition -

How is it working?

LEAP Motion also use infrared sensor to generate depth data, which is useful to recognize objects like human hands or tools like a pen or something like that.

Ákos's Post-It-Player

DEMO TIME

LEAP Motion visualizer

Device limitations

Kinect has a rangeing limit from 0.7 meter to 6 meter.

LEAP Motion: 25mm to 600mm (effective range).

Kinect is good for full body skeleton tracking.

LEAP Motion is good for hand tracking only.

Kinect also capable for hand tracking but not so effective and precise like the LEAP sensor.

Gesture detection

Types of gestures:

static gestures, poses
normal gestures, like waving your hand

Pose detection

Vertical scroll gesture

DEMO TIME - LEAP vertical scroll bookmarklet

DEMO TIME

Vertical scroll with LEAP

User Interface Design with NUI

An intuitive interface is - learnable, understandable, habital.

Beyond intuitiveness things like ergonomics, information design, and reliable gesture detection can make or break your apps success.

Virtual space mapping and interaction resolution

Are people using your application with their elbows resting on the table, seated at a desk with their arms raised, standing at an installation, or something completely different?

There are also situations where the same coordinate mapping may not make sense for each hand. For example, to prevent someone using their right to reach far to the left to access the left side of the screen, the virtual mapping of the right hand can be biased to the right, and vice versa for the left hand.

User Interface Design with NUI

- FEEDBACK

the sensor is working?
what the sensor "sees"?
the sensor "sees" the real user?
what is the dimensions of the sensor?

Little feedback monitor on the top of the app.
Left is a constantly updating depth video about the user, and also a microphone icon, which indicates when the app recognized speech.

Click feedback design with Kinect

Problem: you can't click with Kinect

User Interface Design with NUI

- Menu Interaction designs

Basic menu design - Buttons

Proximity-Based Highlighting

Radial Menus

CodePen example created by Leap Motion team

User Interface Design with NUI

- Menu.Interaction.designs-S01E02

Grid menus

User Interface Design with NUI

- Menu.Interaction.designs-S01E03

Question: how to confirm the choice the user want?

Boundery crossing on X,Y coords

Crossing a boundary in the X,Y plane would be more comfortable and useable than tapping in the air.

Good menu navigation principles:

users can access their options quickly
provide good breadcrumbs so that users know where they are, and how they got there
make it quick to navigate
make it easy to back out or undo a mistaken action