Mind Reading With

Intelligent & Adaptive UIs

@davidkpianoΒ - Agent Conf 2020

🚫 Not adaptive

🚫 Not adaptable

🀬a

🀬a

Everyone is
different.

A/B tests

are ineffective.

User flows are
seldom linear.

πŸ“„

πŸ“„

πŸ“„

πŸ“„

Adaptive user interfaces (AUI)

Adapt to people,
not just devices

β€œIntelligent user interfaces (IUIs) are human-machine interfaces that aim to improve the efficiency, effectiveness, and naturalness of human-machine interaction by representing, reasoning, and acting on models of the user, domain, task, discourse, and media (e.g., graphics, natural language, gesture)”

πŸ“¨ Inbox

πŸ†• VIEW LATEST

βœ‰οΈ New Email

πŸ’Ύ Save as Draft

πŸ“¬ Send

100%

40%

60%

10%

90%

Delete

Are you sure?

Yes

No

Delete

Undo

Deleted!

πŸ‘©β€πŸ’Ό

πŸ™β€β™‚οΈ

πŸ™‹β€β™‚οΈ

Developing AUIs
is challenging.

Data

DATA

Model-driven development

Finite state machines

  • have one initial state

  • a finite number of states

  • a finite number of events

  • a mapping of state transitionsΒ 
    triggered by events

  • a finite number of final states

Sign in

Signing in...

Profile

Home

SIGN IN

SUCCESS

UNAUTHORIZED

PROFILE

Signed Out

SIGN OUT

Signed Out

Statecharts
(Hierarchical State Machines)

import { Machine } from 'xstate';

const feedbackMachine = Machine({
  id: 'feedback',
  initial: 'question',
  states: {
    question: {
      on: {
        CLICK_GOOD: 'thanks',
        CLICK_BAD: 'form',
        CLOSE: 'closed',
        ESC: 'closed'
      }
    },
    form: {
      on: {
        SUBMIT: 'thanks',
        CLOSE: 'closed',
        ESC: 'closed'
      }
    },
    thanks: {
      on: {
        CLOSE: 'closed',
        ESC: 'closed'
      }
    },
    closed: {
      type: 'final'
    }
  }
});
npm install xstate
  • πŸ€– Finite state machines
  • πŸ“₯ Hierarchical states
  • πŸ‘₯ Parallel states
  • πŸ“œ History states
  • πŸ’₯ Interpretation
  • ⏱ Delayed transitions
  • ⏯ Invoked services
  • πŸš€ So much more
import { feedbackMachine } from './feedbackMachine';
import { useMachine } from '@xstate/react';

const App = () => {
  // const [state, dispatch] = useReducer(feedbackReducer, 'question');
  const [current, send] = useMachine(feedbackMachine);

  if (current.matches('question')) {
    return (
      <QuestionScreen
        onClickGood={() => send({ type: 'GOOD' })}
        onClickBad={() => send({ type: 'BAD' })}
        onClose={() => send({ type: 'CLOSE' })}
      />
    );
  } else if (/* ... */) {
    // ...           
  }
}

How was your experience?

π˜…

Good

Bad

Tell me why?

π˜…

Submit

Thanks for your feedback.

π˜…

Close

Ain't nothin' but a heartache

omg conference wifi

  1. Abstract model
  2. Transition analytics
  3. Identify adaptive paths
  4. Use analysis for adaptation

Game plan

Success

Signing in

Error

0.9

0.1

Weighted graphs

State machines inΒ 

VS Live Share

Signed out

Signing in

Signed in

SIGN IN

SIGN IN SUCCESS

SIGN IN FAILURE

Sharing...

Shared

Joining...

Joined

share

share Success

Join Success

Join

Leave

End Collab session

Sign in

How was your experience?

π˜…

Good

Bad

Tell me why?

π˜…

Submit

Thanks for your feedback.

π˜…

Close

Give Feedback

❓

❓

Ain't nothin' but a mistake

Login

Gallery

Profile

Camera

SUCCESS

TAP PROFILE

TAP CAMERA

BACK

BACK

Scroll

Analysis of transitions

Login, Gallery, Gallery

Login, Gallery, Camera

Login, Gallery, Profile

0.90, 0.31

0.90, 0.57

0.90, 0.12

0.90

0.31

0.57

0.13

UNAUTHORIZED

0.10

SUCCESS

Decision trees

Contextual data

  • Registration date
  • Number of friends
  • Posting frequency
  • Location
  • ...etc.

New user?

< 30 days

>= 30 days

Show ad at top

Friend count?

< 100

>= 100

Show more ads

Show less ads

All data

13%

8%

79%

Prediction via
shortest paths

A

B

C

D

E

  • Shortest pathΒ algorithms (Dijkstra, Bellman-Ford, A* search, etc.)
  • AnalyticsΒ provides weights
  • Represents happy paths
  • Can be automatically generated

A

B

C

D

E

import { Machine } from 'xstate';

const myMachine = Machine({
  // ...
});

const myService = interpret(myMachine)
  .onTransition(state => {
    analytics.track({
      target: state.value,
      source: state.history?.value,
      event: sanitize(event),
      timestamp: Date.now()
    });
  })
  .start();

Source

Target

Event

A

B

C

D

E

import { Machine } from 'xstate';
import { getShortestPaths } from '@xstate/graph';

const myMachine = Machine({
  // ...
});

const shortestPaths = getShortestPaths(myMachine);
  • A: A
  • B: A -> B
  • C: A -> B -> C
  • D: A -> D
  • E: A -> D -> E

A

B

C

D

E

import { Machine } from 'xstate';
import { getShortestPaths } from '@xstate/graph';

const myMachine = Machine({
  // ...
});

const shortestPaths = getShortestPaths(myMachine);

0.9

1.0

0.4

0.6

0.1

0.4

0.6

  • A: A
  • B: A -> D -> E -> B
  • C: A -> D -> E -> C
  • D: A -> D
  • E: A -> D -> E

Future ideas for analysis...

πŸ‘©β€πŸ”¬πŸ‘¨β€πŸ”¬

A

B

C

0.75

0.25

D

E

(D) 0.95

(D) 0.05

A -> B
A -> C
A -> B
A -> B
E ->
D ->
E ->
F ->

Higher-order Markov models

Deep Reinforcement
Learning

-

-

+

+

+

?

Agent

Environment

Action

State

Reward

Deep Reinforcement Learning

Policy

State

Action

❔

Wait... Seriously?

Let's see a demo.

(Show this slide in case the demo fails, which it probably will)

service.onTransition(state => {
  analytics.track({
    source: state.history.value,
    target: state.value,
    context: state.context,
    event: sanitize(state.event),
    timestamp: Date.now()
  })
});

Application

Executable Model

Metrics Tracking

Adaptive Transitions

Adaptive UI

Analysis

➑️ Determinism

πŸ“ˆ Visualization

πŸ‘©β€πŸ’» DEV Experience

🀝 Communication

πŸ“Š Analytics

πŸ‘©β€πŸ”¬ Simulation

πŸ”¬ Testability

πŸ‘¨β€πŸ‘©β€πŸ‘§β€πŸ‘¦ User Experience

xstate
@xstate/react
@xstate/graph
@xstate/vue
@xstate/test
@xstate/fsm
@xstate/analytics
@xstate/viz

βœ…

βœ…

βœ…

πŸ”œ

πŸ”œ

βœ…

πŸ”œ

βœ…

npm install xstate

@xstate/scxml

πŸ”œ

@xstate/immer

πŸ”œ

xstate.js.org/viz

The world of statecharts

Spectrum community

XState Docs

Make your code do more.

Thank you Agent Conf!

@davidkpianoΒ - Agent Conf 2020

Mind Reading with Intelligent & Adaptive UIs

By David Khourshid

Mind Reading with Intelligent & Adaptive UIs

Agent Conf 2020

  • 3,350