Help!

I've Fallen (into code), and I Can't Get up!

How + why to make frontend maps

Cameron Yick | @hydrosquall

data vis engineer @ Datadog

About

Past
- ⚡️ Electrical Engineering - Circuits
- 🚿 Data Engineering - Pipelines
Present
- Frontend data visualization
  - 🌞: Interactions + Topology Maps at Datadog
  - 🌖 : React for visual remakes

remake historical visualizations (Train schedule)

NYC take on the cover graphic of Edward Tufte's Visual Display of Quantitative Information

git/Trello history

FEELING LOST?

lost (in code)

Join project / team
npm install new library
Investigating bug while on call
Visit code from 1 month (or hour) ago

getting unlost in the real world

Wander
- Landmarks / signs
Directions
- local vs fellow tourist
Map
- GPS (You are here)
- Visual route

getting unlost in the Real code World

Wander (scroll + ctrl+f)
- Landmarks: (entrypoint, data structures)
- Signs: comments / README
- Signal flares: breakpoints / console.log
Directions
- ```
git blame / CODEOWNERS
```

Map
- Luck: Handmade diagram

maps: the first data visualizations

...graphical representations that facilitate spatial understanding of things... 

- Harley & Woodward, History of Cartography

maps are abstract (!🐛)

"Not only is it easy to lie with maps, it's essential...

To avoid hiding critical information in a fog of detail, the map must offer a selective, incomplete view of reality"

- Mark Monmonier, "How to Lie With Maps"

the ideal

abstraction level

situation specific.

WHEN (FIXING BUGS | ADDING FEATURES)

which areas do I need to understand / change?

maps & unknown unknowns: Dragons

Marine Map of Scandinavia: 1539

today's plan

past
- intro to a research tool
present
- tools we can use now
future
- making tools beyond maps

PAST

Present
Future

A technique from 18 Years B.R. (Before React)

the eternal question

How can we keep the plan and the implementation in sync?

"Software Reflexion Models" (1995)

Peter Bergstrom's PaperCube (Citation Graph Browser)

(Manual) Make high level concept model
(Manual) Associate concepts with source code
(Automatic) Generate automatic source code relationships
- Imports, call graphs, data flow
(Automatic) Combine
- Add mapping (2) and relationships (3) to the concept model (1)
- Visualize the difference

stages of reflexion

the differences are where the bugs hide!

"Connecting task to source" (Slides 2014)

Data API

Container

View

1. Container/View concept model

Imported by

React Reflexion example

// data-service.js
export const fetchItems = async () => fetch('/api/items');

// List.jsx
import { formatItems } from './data-service';
const formatItems = (items) = items.map(formatItem);

export const List = (props) => {
  return (
  <div>
    {formatItems(props.items)}
  </div>
  )
}

// ListContainer.jsx
import { fetchItems } from './data-service';

export const ListContainer = (props) => {
  // pseudocode for data fetching
  const items = await fetchItems(); 
  return <List items={items}/>
}

2. Map Concepts to files

data-api: *-service.js
container: *Container.jsx
view: *.jsx

3. run madge

Data API

Container

View

everything is matches!

Matching

Missing

Unexpected

2 sprints later...

```
Added /people API 
```
"Refactoring"
All unit tests still pass

// data-service.js
export const fetchItems = async () => fetch('/api/items');
export const formatItems = (items) => items.map(item => processItem(item));

// Someone unfamiliar with project structure proposes a new change
export const fetchPeople = async () => fetch('/api/people');

// List.jsx
import { formatItems, fetchItems, fetchPeople } from './data-service';

export const List = (props) => {
  const items = await fetchItems();
  const people = await fetchPeople(); 
  
  return (
  <div>
    {formatItems(items)}
    {people}
  </div>
  )
}

// ListContainer.jsx
export const ListContainer = (props) => {
  // some other code for logging, routing etc
  return <List />
}

changes

Data API

Container

View

post refactor reflexion

Matching

Missing

Unexpected

Refactored

New import

Data API - model realignment?

Deciding the fate of Data API:
- Is it for HTTP, transformation, or both?
Change the code
- Move functions around
Change the concept
- Rename Data to be more specific
- Make separate blocks for HTTP/Transform

"Connecting task to source" (Slides 2014)

question Revisited

How can we keep the plan and implementation in sync?

reflexion takeaways

Model and code can both be "right" (map & territory blur) as needs change over time
People + machines are good at different things, effective tools let each focus on their strengths

PAST

Present
Future

Tools usable now

how can we keep maps + code in sync today?

signal processing +

visual programming

MaxMSP (audio)

meemoo - web-based graphics

Touchdesigner: 3d Graphics

visual / node programming 🐰🕳

github.com/ivanreese/visual-programming-codex

We don't have this for JS

(and this won't help our existing applications)

what (useful) visuals can be generated from code?

DG + DAG

(Directed Acyclic Graphs)

A B

B "uses" A

what do nodes represent?

Continent (Eurasia)

Country

Subnational (State, Province, County)

City

Neighborhood

Ladder of geoPolitical Abstraction

Specific

Broader

it's not about size

UK (country) projected onto Texas (US State)

Imports (module/file)

Call graphs (function)

Dataflow (variable)

Down the Ladder of code Abstraction (for Nodes)

Specific

Abstract

Others: State, "Components"

module imports

const node = "file"
`Which ${node} might be affected 
 by changes to other ${node}s?`

ARKit

https://arkit.pro

ARKit: conceptual Groups

https://arkit.pro

See Github for Client/Server example

dependency-cruiser: folders

https://www.netlify.com/blog/2018/08/23/how-to-easily-visualize-a-projects-dependency-graph-with-dependency-cruiser/

madge: COlors + extensible API

https://github.com/pahen/madge

FUnction calL

GRAPHS

const node = "function"
`Which ${node} might be affected 
 by changes to other ${node}s?`

*Not necessarily DAG (Recursion / Loops)

dynamic callgraph

Record of actual run (single path though tree)

STATIC CALLGRAPH 🚧

(Python Example)

persper/JS-callgraph 🚧

Have data, seeking visualization

datafloW (variables)

const node = "variable"
`Which ${node} might be affected 
 by changes to other ${node}s?`

EXCEL + js + jupyter = observable

reactive js Notebook

Observable Dataflow

https://observablehq.com/@mbostock/notebook-visualizer

Reselect-toolS

https://github.com/skortchmark9/reselect-tools

- Reposition/Highlight upstream / downstream

- Inspect runtime values

- Count recomputations (hotspots)

reselect @ runtime

State machines

Automatic visual + runnable documentation

const pedestrianStates = {
  initial: 'walk',
  states: {
    walk: {
      on: {
        PED_TIMER: 'wait'
      }
    },
    wait: {
      on: {
        PED_TIMER: 'stop'
      }
    },
    stop: {}
  }
};
const lightMachine = Machine({
  id: 'light',
  initial: 'green',
  states: {
    green: {
      on: {
        TIMER: 'yellow'
      }
    },
    yellow: {
      on: {
        TIMER: 'red'
      }
    },
    red: {
      on: {
        TIMER: 'green'
      },
      ...pedestrianStates
    }
  }
});
// via xstate docs page

Xstate traffic light

See @DavidKPiano's XState talks

light
  green
    timer -> yellow
  yellow
    timer -> red
  red
    walk
      ped_timer -> wait
    wait
     ped_timer -> stop
    stop
     timer -> green

Sketch.systems -> Xstate

- Generate XState JSON from shorthand

- Runnable automatic visual documentation

PAST

Present
Future

ideas to make existing maps more useful

2 Challenges + approaches

1. Hairball -> Visual Search Mantra

2. Wayfinding -> Adding Data

graph visual noise is not a new problem

Future: Biofabric / Hive Plots / Hyperbolic Tree / other non-mainstream layouts

Why managing hairballs matters

Source: EagerEyes / It's Always Sunny in Philadelphia

show everything,

notice nothing

- not a real Yoda quote

🗺: we don't need to show everything.

Source: AxisMaps

information seeking mantra

Ben "Hyperlink" Schneiderman (The Eyes Have It, 1996)

1. overview first

2. Zoom + Filter

3. Details on Demand

mantra applied

1. overview first

2. Zoom + Filter

3. Details on Demand

Automatic summary

Visual pruning

Click for detail

demo electron app

Examining JSNetworkX + nteract

1. overview: madge + interactions + color

2. Filtering: prune Files/Folders

2. Filtering: undo/refine

Filtering enables alternate layouts

3: details on demand: neighbors

3: details... git metadata

wayfinding: annotating space

- Architecture information systems (signage)

- Help people to orient themselves

- Enhance understanding + experience of complex spaces

@Logan Airport

core Wayfinding questions

Orientation
- Where am I?
Selection
- Where should I go?
Monitoring
- Am I on track?
Destination ID
- Are we there yet?

just add water data

political maps are just the beginning (base Layer)

Weather

Flight Routes

node success status (Airflow)

Function usage (python)

Reselect-tools makes these too

put search results on a map!

1. Pick a search tool

(ripgrep, Sourcegraph, etc)

2. Pick a map (filesystem, any directed graph)

3. Use 2 to organize the results from 1

search demo: finding #todos

JSNetworkx/src/algorithms

include details + interaction

PAST

Present
Future

maps- not just for the real world

Recap

Highlight plan divergence / use human + machine pairings
Automatic tools exist for all rungs of the abstraction ladder
Info seeking mantra + wayfinding helps manage busyness

We shape our tools and thereafter our tools shape us

- Attributed to Marshall McLuhan

Thank you!

Keen to hear about your {most|least} favorite diagrams/maps!

Twitter: @hydrosquall

Pronouns: he/him

Demos/More Reading: github.com/hydrosquall/code-maps-frontend

Datadog: We're hiring!

What might we achieve with tools that help us understand code more quickly and accurately?

This page intentionally left blank. Please do not write on it.

demo built with

Electron
- Electron React Boilerplate
- Hot reloaded server: James Long's electron-with-server
Graph visualization / analysis
- Vis.js
- Cytoscape + dagre layout
- JSNetworkX
Data
- Madge - module dependencies
- node-gitlog - git history
- directory-tree
- ripgrep: search
Material-ui, dis.gui, immer

Help!

I've Fallen (into code), and I Can't Get up!

Cameron Yick | @hydrosquall

data vis engineer @ Datadog

About

remake historical visualizations (Train schedule)

git/Trello history

FEELING LOST?

lost (in code)

getting unlost in the real world

getting unlost in the Real code World

maps: the first data visualizations

maps are abstract (!🐛)

WHEN (FIXING BUGS | ADDING FEATURES)

which areas do I need to understand / change?

maps & unknown unknowns: Dragons

today's plan

PAST

Present Future

A technique from 18 Years B.R. (Before React)

the eternal question

"Software Reflexion Models" (1995)

stages of reflexion

the differences are where the bugs hide!

"Connecting task to source" (Slides 2014)

1. Container/View concept model

React Reflexion example

2. Map Concepts to files

3. run madge

everything is matches!

2 sprints later...

changes

post refactor reflexion

Data API - model realignment?

"Connecting task to source" (Slides 2014)

question Revisited

reflexion takeaways

PAST

Present Future

Tools usable now

how can we keep maps + code in sync today?

signal processing +

visual programming

MaxMSP (audio)

meemoo - web-based graphics

Touchdesigner: 3d Graphics

visual / node programming 🐰🕳

what (useful) visuals can be generated from code?

DG + DAG

what do nodes represent?

Ladder of geoPolitical Abstraction

it's not about size

Down the Ladder of code Abstraction (for Nodes)

module imports

ARKit

ARKit: conceptual Groups

dependency-cruiser: folders

madge: COlors + extensible API

FUnction calL

GRAPHS

dynamic callgraph

STATIC CALLGRAPH 🚧

persper/JS-callgraph 🚧

datafloW (variables)

EXCEL + js + jupyter = observable

reactive js Notebook

Observable Dataflow

Reselect-toolS

reselect @ runtime

State machines

Xstate traffic light

Sketch.systems -> Xstate

PAST

Present Future

ideas to make existing maps more useful

2 Challenges + approaches

graph visual noise is not a new problem

Why managing hairballs matters

show everything,

notice nothing

Present
Future

Present
Future

Present
Future

Present
Future