Mind Reading With

Intelligent & Adaptive UIs

@davidkpiano - Agent Conf 2020

🚫 Not adaptive

🚫 Not adaptable

🤬a

Everyone is
different.

A/B tests

are ineffective.

User flows are
seldom linear.

guess-js.github.io

📄

github.com/GoogleChromeLabs/react-adaptive-hooks

Adaptive user interfaces (AUI)

Adapt to people,
not just devices

“Intelligent user interfaces (IUIs) are human-machine interfaces that aim to improve the efficiency, effectiveness, and naturalness of human-machine interaction by representing, reasoning, and acting on models of the user, domain, task, discourse, and media (e.g., graphics, natural language, gesture)”

(Maybury, 1999)

📨 Inbox

🆕 VIEW LATEST

✉️ New Email

💾 Save as Draft

📬 Send

100%

40%

60%

10%

90%

Delete

Are you sure?

Yes

Delete

Undo

Deleted!

IntelliCode for VS Code

👩‍💼

🙍‍♂️

🙋‍♂️

Developing AUIs
is challenging.

Source: Current Challenges and Applications for Adaptive User Interfaces

Data

DATA

Model-driven development

Source: https://dribbble.com/shots/6206737-Early-stage-flowcharts-for-Mobile-users

Finite state machines

have one initial state
a finite number of states
a finite number of events
a mapping of state transitions
triggered by events
a finite number of final states

Signing in...

Profile

Home

SUCCESS

UNAUTHORIZED

PROFILE

Signed Out

SIGN OUT

Signed Out

Statecharts
(Hierarchical State Machines)

Statecharts: A Visual Formalism for Complex Systems (David Harel, 1987)

import { Machine } from 'xstate';

const feedbackMachine = Machine({
  id: 'feedback',
  initial: 'question',
  states: {
    question: {
      on: {
        CLICK_GOOD: 'thanks',
        CLICK_BAD: 'form',
        CLOSE: 'closed',
        ESC: 'closed'
      }
    },
    form: {
      on: {
        SUBMIT: 'thanks',
        CLOSE: 'closed',
        ESC: 'closed'
      }
    },
    thanks: {
      on: {
        CLOSE: 'closed',
        ESC: 'closed'
      }
    },
    closed: {
      type: 'final'
    }
  }
});

github.com/davidkpiano/xstate

npm install xstate

xstate.js.org

🤖 Finite state machines
📥 Hierarchical states
👥 Parallel states
📜 History states
💥 Interpretation
⏱ Delayed transitions
⏯ Invoked services
🚀 So much more

import { feedbackMachine } from './feedbackMachine';
import { useMachine } from '@xstate/react';

const App = () => {
  // const [state, dispatch] = useReducer(feedbackReducer, 'question');
  const [current, send] = useMachine(feedbackMachine);

  if (current.matches('question')) {
    return (
      <QuestionScreen
        onClickGood={() => send({ type: 'GOOD' })}
        onClickBad={() => send({ type: 'BAD' })}
        onClose={() => send({ type: 'CLOSE' })}
      />
    );
  } else if (/* ... */) {
    // ...           
  }
}

How was your experience?

𝘅

Good

Bad

Tell me why?

𝘅

Submit

Thanks for your feedback.

𝘅

Ain't nothin' but a heartache

https://codepen.io/davidkpiano/pen/dc81af7260581c1fbbf5b5154caa2228

omg conference wifi

Abstract model
Transition analytics
Identify adaptive paths
Use analysis for adaptation

Game plan

Success

Signing in

Error

0.9

0.1

Weighted graphs

State machines in

VS Live Share

Signed out

Signing in

Signed in

Sharing...

Shared

Joining...

Joined

share Success

Join Success

Join

Leave

End Collab session

How was your experience?

𝘅

Good

Bad

Tell me why?

𝘅

Submit

Thanks for your feedback.

𝘅

Give Feedback

❓

Ain't nothin' but a mistake

Gallery

Profile

Camera

SUCCESS

TAP PROFILE

TAP CAMERA

BACK

Scroll

Analysis of transitions

0.90, 0.31

0.90, 0.57

0.90, 0.12

0.90

0.31

0.57

0.13

UNAUTHORIZED

0.10

SUCCESS

Decision trees

Contextual data

Registration date
Number of friends
Posting frequency
Location
...etc.

New user?

< 30 days

>= 30 days

Show ad at top

Friend count?

< 100

>= 100

Show more ads

Show less ads

More info: Towards Data Science -
Decision Trees in Machine Learning

All data

13%

79%

Prediction via
shortest paths

Shortest path algorithms (Dijkstra, Bellman-Ford, A* search, etc.)
Analytics provides weights
Represents happy paths
Can be automatically generated

import { Machine } from 'xstate';

const myMachine = Machine({
  // ...
});

const myService = interpret(myMachine)
  .onTransition(state => {
    analytics.track({
      target: state.value,
      source: state.history?.value,
      event: sanitize(event),
      timestamp: Date.now()
    });
  })
  .start();

Source

Target

Event

import { Machine } from 'xstate';
import { getShortestPaths } from '@xstate/graph';

const myMachine = Machine({
  // ...
});

const shortestPaths = getShortestPaths(myMachine);

```
A: A
```
```
B: A -> B
```
```
C: A -> B -> C
```
```
D: A -> D
```
```
E: A -> D -> E
```

import { Machine } from 'xstate';
import { getShortestPaths } from '@xstate/graph';

const myMachine = Machine({
  // ...
});

const shortestPaths = getShortestPaths(myMachine);

0.9

1.0

0.4

0.6

0.1

0.4

0.6

```
A: A
```
```
B: A -> D -> E -> B
```
```
C: A -> D -> E -> C
```
```
D: A -> D
```
```
E: A -> D -> E
```

Future ideas for analysis...

👩‍🔬👨‍🔬

0.75

0.25

(D) 0.95

(D) 0.05

A -> B
A -> C
A -> B
A -> B

E ->
D ->
E ->
F ->

Higher-order Markov models

Variable-order Markov models (Wikipedia)

Deep Reinforcement
Learning

Agent

Environment

Action

State

Reward

Deep Reinforcement Learning

Policy

State

Action

https://medium.com/coinmonks/the-artificial-neural-networks-handbook-part-1-f9ceb0e376b4

❔

Wait... Seriously?

Let's see a demo.

Towards Data Science article on RL with Python

(Show this slide in case the demo fails, which it probably will)

service.onTransition(state => {
  analytics.track({
    source: state.history.value,
    target: state.value,
    context: state.context,
    event: sanitize(state.event),
    timestamp: Date.now()
  })
});

Application

Executable Model

Metrics Tracking

Adaptive Transitions

Adaptive UI

Analysis

➡️ Determinism

📈 Visualization

👩‍💻 DEV Experience

🤝 Communication

📊 Analytics

👩‍🔬 Simulation

🔬 Testability

👨‍👩‍👧‍👦 User Experience

xstate

@xstate/react

@xstate/graph

@xstate/vue

@xstate/test

@xstate/fsm

@xstate/analytics

@xstate/viz

✅

🔜

✅

🔜

xstate.js.org/docs

statecharts.github.io

spectrum.chat/statecharts

✅

npm install xstate

@xstate/scxml

🔜

@xstate/immer

🔜

xstate.js.org/viz

The world of statecharts

statecharts.github.io

Spectrum community

spectrum.chat/statecharts

XState Docs

xstate.js.org/docs

Make your code do more.

Thank you Agent Conf!

@davidkpiano - Agent Conf 2020

Mind Reading With

Intelligent & Adaptive UIs

Everyone is different.

A/B tests

are ineffective.

User flows are seldom linear.

Adaptive user interfaces (AUI)

Adapt to people, not just devices

📨 Inbox

🆕 VIEW LATEST

✉️ New Email

💾 Save as Draft

📬 Send

Developing AUIs is challenging.

Model-driven development

Finite state machines

Statecharts(Hierarchical State Machines)

Game plan

Weighted graphs

State machines in

VS Live Share

Analysis of transitions

Decision trees

Prediction via shortest paths

Future ideas for analysis...

Higher-order Markov models

Deep Reinforcement Learning

Deep Reinforcement Learning

Wait... Seriously?

➡️ Determinism

📈 Visualization

👩‍💻 DEV Experience

🤝 Communication

📊 Analytics

👩‍🔬 Simulation

🔬 Testability

👨‍👩‍👧‍👦 User Experience

The world of statecharts

statecharts.github.io

Spectrum community

spectrum.chat/statecharts

XState Docs

xstate.js.org/docs

Make your code do more.

Thank you Agent Conf!

Everyone is
different.

User flows are
seldom linear.

Adapt to people,
not just devices

Developing AUIs
is challenging.

Statecharts
(Hierarchical State Machines)

Prediction via
shortest paths

Deep Reinforcement
Learning