Mind Reading With

Intelligent & Adaptive UIs

@davidkpiano ยท JSDC Taiwan 2019

Fetch data

๐Ÿ“„

GET api/users

Pending

200 OK

Fetch data

๐Ÿ“„

GET api/users

Pending

200 OK

Fetch data

๐Ÿ“„

GET api/users

Pending

๐Ÿ“„

GET api/users

Pending

๐Ÿ“„

GET api/users

Pending

๐Ÿ“„

GET api/users

Pending

๐Ÿ“„

GET api/users

Pending

Fetch data

๐Ÿ“„

GET api/users

Pending

200 OK

Fetch data

๐Ÿ“„

GET api/users

Pending

๐Ÿ“„

GET api/users

Pending

๐Ÿ“„

GET api/users

Pending

๐Ÿ“„

GET api/users

Pending

Fetch data

Everyone is
different.

A/B tests

are ineffective.

User flows are
seldom linear.

๐Ÿ“„

๐Ÿ“„

๐Ÿ“„

๐Ÿ“„

Adaptive user interfaces (AUI)

Adapt to people,
not just devices

๐Ÿ“จ Inbox

๐Ÿ†• VIEW LATEST

โœ‰๏ธ New Email

๐Ÿ’พ Save as Draft

๐Ÿ“ฌ Send

100%

40%

60%

10%

90%

Delete

Are you sure?

Yes

No

Delete

Undo

Deleted!

๐Ÿ‘ฉโ€๐Ÿ’ผ

๐Ÿ™โ€โ™‚๏ธ

๐Ÿ™‹โ€โ™‚๏ธ

Developing AUIs
is challenging.

Model-driven development

Finite state machines

  • have one initial state

  • a finite number of states

  • a finite number of events

  • a mapping of state transitionsย 
    triggered by events

  • a finite number of final states

Sign in

Signing in...

Profile

Home

SIGN IN

SUCCESS

UNAUTHORIZED

PROFILE

Signed Out

SIGN OUT

Signed Out

Define transitions between
states & events

function searchReducer(state = 'idle', event) {
  switch (state) {
    case 'idle':
      switch (event.type) {
        case 'SEARCH':
          return 'searching';
        default:
          return state;
      }
    case 'searching':
      switch (event.type) {
        case 'RESOLVE':
          return 'success';
        case 'REJECT':
          return 'failure';
        default:
          return state;
      }
    case 'success':
      switch (event.type) {
        case 'SEARCH':
          return 'searching';
        default:
          return state;
      }
    case 'failure':
      switch (event.type) {
        case 'SEARCH':
          return 'searching';
        default:
          return state;
      }
    default:
      return state;
  }
}

FSMs with switch/case

const machine = {
  id: 'feedback',
  initial: 'question',
  states: {
    question: {
      on: {
        CLICK_GOOD: 'thanks',
        CLICK_BAD: 'form',
        CLOSE: 'closed',
        ESC: 'closed'
      }
    },
    form: {
      on: {
        SUBMIT: 'thanks',
        CLOSE: 'closed',
        ESC: 'closed'
      }
    },
    thanks: {
      on: {
        CLOSE: 'closed',
        ESC: 'closed'
      }
    },
    closed: {
      type: 'final'
    }
  }
};

Define transitions between
states & events

const transition = (state, event) => {
  return machine
    .states[state] // current state
    .on[event]     // next state
    || state;      // or same state
}

FSMs with object mapping

import { Machine } from 'xstate';

const feedbackMachine = Machine({
  id: 'feedback',
  initial: 'question',
  states: {
    question: {
      on: {
        CLICK_GOOD: 'thanks',
        CLICK_BAD: 'form',
        CLOSE: 'closed',
        ESC: 'closed'
      }
    },
    form: {
      on: {
        SUBMIT: 'thanks',
        CLOSE: 'closed',
        ESC: 'closed'
      }
    },
    thanks: {
      on: {
        CLOSE: 'closed',
        ESC: 'closed'
      }
    },
    closed: {
      type: 'final'
    }
  }
});
npm install xstate
import { feedbackMachine } from './feedbackMachine';
import { useMachine } from '@xstate/react';

const App = () => {
  // const [state, dispatch] = useReducer(feedbackReducer, 'question');
  const [current, send] = useMachine(feedbackMachine);

  if (current.matches('question')) {
    return (
      <QuestionScreen
        onClickGood={() => send({ type: 'GOOD' })}
        onClickBad={() => send({ type: 'BAD' })}
        onClose={() => send({ type: 'CLOSE' })}
      />
    );
  } else if (/* ... */) {
    // ...           
  }
}

Statecharts
(Hierarchical State Machines)

Statecharts

Idle

Searching...

entry / prefetchResources

entry / fetchResults

Search

  • Actionsย - onEntry, onExit, transition
  • Guardsย - conditional transitions

[query.length > 0]

Statecharts

  • Actionsย - onEntry, onExit, transition
  • Guardsย - conditional transitions
  • Hierarchyย - nested states
  • Orthogonalityย - parallel states
  • Historyย - remembered states

H

Idle

Searching...

SEARCH

Success

Failure

RESOLVE

REJECT

SEARCH

Searched

Hierarchical states

Idle

Searching...

SEARCH

Success

Failure

RESOLVE

REJECT

SEARCH

Searched

Hierarchical states

const machine = Machine({
  initial: 'idle',
  states: {
    // ...
    searching: {
      on: {
        RESOLVE: 'searched',
        REJECT: 'searched.failure'
      }
    },
    searched: {
      initial: 'success',
      states: {
        success: {},
        failure: {}
      },
      on: { SEARCH: 'searching' }
    }
  }
});

How was your experience?

๐˜…

Good

Bad

Tell me why?

๐˜…

Submit

Thanks for your feedback.

๐˜…

Close

Ain't nothin' but a heartache

  1. Abstract model
  2. Transition analytics
  3. Identify adaptive paths
  4. Use analysis for adaptation

Game plan

Photo by NEW DATA SERVICESย on Unsplash

Success

Signing in

Error

0.9

0.1

Weighted graphs

State machines inย 

VS Live Share

Signed out

Signing in

Signed in

SIGN IN

SIGN IN SUCCESS

SIGN IN FAILURE

Sharing...

Shared

Joining...

Joined

share

share Success

Join Success

Join

Leave

End Collab session

Sign in

How was your experience?

๐˜…

Good

Bad

Tell me why?

๐˜…

Submit

Thanks for your feedback.

๐˜…

Close

Give Feedback

โ“

โ“

Ain't nothin' but a mistake

Login

Gallery

Profile

Camera

SUCCESS

TAP PROFILE

TAP CAMERA

BACK

BACK

Scroll

Analysis of transitions

Login, Gallery, Gallery

Login, Gallery, Camera

Login, Gallery, Profile

0.90, 0.31

0.90, 0.57

0.90, 0.12

0.90

0.31

0.57

0.13

UNAUTHORIZED

0.10

SUCCESS

(decide)

TAP AD

Analysis of events

+10

Gallery

Ads interspersed

Ads shown at top

Ad

Scroll

Scroll

TAP AD

+10

-1

-10

INTERSPERSED > .5

TOP > .5

Decision trees

Contextual data

  • Registration date
  • Number of friends
  • Posting frequency
  • Location
  • ...etc.

New user?

< 30 days

>= 30 days

Show ad at top

Friend count?

< 100

>= 100

Show more ads

Show less ads

All data

13%

8%

79%

Prediction via
shortest paths

A

B

C

D

E

  • Shortest pathย algorithms (Dijkstra, Bellman-Ford, A* search, etc.)
  • Analyticsย provides weights
  • Represents happy paths
  • Can be automatically generated

A

B

C

D

E

import { Machine } from 'xstate';

const myMachine = Machine({
  // ...
});

const myService = interpret(myMachine)
  .onTransition(state => {
    analytics.track({
      target: state.value,
      source: state.history?.value,
      event: sanitize(event),
      timestamp: Date.now()
    });
  })
  .start();

Source

Target

Event

A

B

C

D

E

import { Machine } from 'xstate';
import { getShortestPaths } from '@xstate/graph';

const myMachine = Machine({
  // ...
});

const shortestPaths = getShortestPaths(myMachine);
  • A: A
  • B: A -> B
  • C: A -> B -> C
  • D: A -> D
  • E: A -> D -> E

A

B

C

D

E

import { Machine } from 'xstate';
import { getShortestPaths } from '@xstate/graph';

const myMachine = Machine({
  // ...
});

const shortestPaths = getShortestPaths(myMachine);

0.9

1.0

0.4

0.6

0.1

0.4

0.6

  • A: A
  • B: A -> D -> E -> B
  • C: A -> D -> E -> C
  • D: A -> D
  • E: A -> D -> E

A

B

C

0.75

0.25

D

E

(D) 0.95

(D) 0.05

A -> B
A -> C
A -> B
A -> B
E ->
D ->
E ->
F ->

Higher-order Markov models

Deep Reinforcement
Learning

-

-

+

+

+

Agent

Environment

Action

State

Reward

Deep Reinforcement Learning

service.onTransition(state => {
  analytics.track({
    source: state.history.value,
    target: state.value,
    context: state.context,
    event: state.event,
    timestamp: Date.now()
  })
});

Application

Executable Model

Metrics Tracking

Adaptive Transitions

Adaptive UI

โžก๏ธ Determinism

๐Ÿ“ˆ Visualization

๐Ÿ‘ฉโ€๐Ÿ’ป DEV Experience

๐Ÿค Communication

๐Ÿ“Š Analytics

๐Ÿ‘ฉโ€๐Ÿ”ฌ Simulation

๐Ÿ”ฌ Testability

๐Ÿ‘จโ€๐Ÿ‘ฉโ€๐Ÿ‘งโ€๐Ÿ‘ฆ User Experience

๐Ÿšš Prevent Information Overload

๐Ÿ’ก Provide intuitive guidance

โœ… Understand Tasks & Goals

โš™๏ธ Create Personalized Systems

Adaptive User Interfaces

๐Ÿ”€ Micro vs. Macro Adaptivity

โš™๏ธ Prediction vs. Feedback

๐ŸŒ Local vs. Global

๐Ÿ“Š Data vs. Simulation

Modeling considerations

Advantages

of using statecharts

  • Visualized modeling
  • Precise diโ€‹agrams
  • Automatic code generation
  • Comprehensive test coverage
  • Accommodation of late-breaking requirements changes

Disadvantages

of using statecharts

Learning curve

Modeling requires planning ahead

Not everything can be modeled (yet)

Statecharts

FSMs

Bottom-up

Complexity

trade-offs

States & logic

Code Complexity

xstate
@xstate/react
@xstate/graph
@xstate/vue
@xstate/test
@xstate/analytics
@xstate/sim
@xstate/viz

โœ…

โœ…

โœ…

๐Ÿ”œ

๐Ÿ”œ

๐Ÿ”œ

๐Ÿ”œ

โœ…

The world of statecharts

Spectrum community

XState Docs

Make your code do more.

Thank you JSDC!

@davidkpianoย ยท JSDC Taiwan 2019

๐Ÿ”ฎ

Mind Reading with Intelligent & Adaptive UIs

By David Khourshid

Private

Mind Reading with Intelligent & Adaptive UIs

JSDC 2019