Data Stream Correlation

Evan Dana | https://github.com/evandana/stream-correlation

Overwhelming

Actionable

  1. Scenario

  2. Challenges

  3. Demo

Scenario

Given a continuous stream of data for multiple series

When I focus on one stream

Then I want to see which streams are correlated

Challenges

Technology agnostic implementation considerations comprise the most meaningful and lasting benefit from this exercise.

  • How much data should be analyzed to determine correlation?
    • Count of series?
    • Data point count for significant correlation?
  • How does the data structure translate from initial series to many multi-thread series?
  • How do you handle two series with intermittent correlation?
  • Efficient pub/sub patterns
  • Scalability
  • Succinct developer initialization command

Data Flow

pub/sub

pub/sub

Generate Data

Service

Process Data

Service

UI Visualization

Client

Correlation Library: https://www.npmjs.com/package/correlation-rank

Demo

Only available when running app locally

Questions?

Thanks!

github: https://github.com/evandana/stream-correlation

Evan Dana

Made with Slides.com