Data Mining Overview
Tables
Hypercubes
Metacube
Tables
Goals
- Data mining tables aggregate information from many feature tables to provide insight or answer a question
- Data mining tables are used to create hypercubes for analyzing experiments
Features
- Data mining tables do not affect the recommendation system in production
- Data mining tables are used for analysis of features during a sliding 90-day timeframe
Extract
Transform
Load
Extract
Transform
Load
Load to Redshift
Workflow Objects
- Inherits from:
- hive_workflow
- hive_daily_workflow
- Option to upload to Redshift
- Custom validation logic
Successes
- Easy to create a workflow
- Configurable
- Upstream validation
- Terminal validation
Regrets
- Latency in extract phase
- Source of truth is Redshift
- String transform in load to Redshift
- Formatting difficulties
- Many workflow objects
- Duplicated tests
The Future
Table Name
Schedule
Frequency
Dependencies
Validation
Hypercubes
Goals
- Hypercubes allow slicing, or dimension reduction
- Hypercubes allow drill-up/drill down; you can analyze one dimension or increase complexity by examining interaction between many dimensions
- Hypercubes allow roll up; you can summarize across one dimension
Features
- Hypercubes can be configured on the fly (CLI) or in a more permanent fashion (hypercubes.py)
- Hypercubes are self-building
- Hypercube queries are optimized for performance
- Hypercubes are shareable and can be combined
Hypercube Components
- Hypercube map
- Hypercube core
- Feature Hypercube
- Experiment Hypercube
Successes
- Configurable
- Componetized
Regrets
- High latency building multiple hypercubes, even with multiprocessing
The Future
Component
Experiment
Hashing
Measures
Features
Denominators
Metacube
Overview
The metacube combines all of the experiment hypercubes into one table for uploading to redshift, for use in Tableau, R, & Chartio.
Experiment Workbook
