Agile A.I.

Bridging the Gap Between Product Management and Data Science

About Me

Garrett Eastham

edgecase

Founder & Chief Data Scientist

• AI & Ecommerce Focus

• CS @ Stanford

• Background in Web Analytics

• Career in Product Management

• Prior: Bazaarvoice, RetailMeNot

Today's Talk

The Gap as it Exists Today

Conducting Agile Data Science

Running an AI Playbook

Example: Customer Service Optimization

The Gap

Defining Machine Intelligence

Machine Intelligence (n): the power of a machine to copy intelligent human behavior.

Theory

Reality

Machine Intelligence (n): enabling digital experiences to leverage trained and/or derived statistical models for the purpose of extending an existing or new value proposition in manners not possible prior.

Where is the Gap?

Product Management

Data Science

Data & Market Opp.

Delivered Value to Customer

?

Two Types of People

Data Scientist

Product Manager

• Customer-focused
• Abstract value from implementation

• Data-focused
• Work requires extreme focus to detail

Common Stakeholder Questions

"We have lots of data, but I have no idea how to make use of it - or if it's even worth it."

"I've heard great things about AI, but I have no idea what's even possible for us."

"How do we get to an understanding of cost and scope for something no one understands."

Conducting Agile Data Science

What Makes it Agile?

Individuals and Interactions

over processes and tools

Working Software

over comprehensive documentation

Customer Collaboration

over contract negotiation

Responding to Change

over following a plan

The Agile AI Framework

Goal: Optimize the necessary cross-functional collaboration and coordinated output of product teams attempting to create value through machine intelligence

Front-load "necessary" data plumbing activities to enable use case exploration

Allow value-propositions to be rooted in actual data without sacrificing creativity

Enable non-technical stakeholders to participate at all phases of development

The Agile AI Framework

Business Development

Subject Matter Expertise

Data Science & Engineering

User Experience

Product Realization

Non-Linear Collaboration

Linear Output

The Agile AI Framework

Business Development

User Experience

Subject Matter Expertise

Product Realization

Data Science & Engineering

Data Plumbing & Exploration

Model Development

Model Scaling & Packaging

ROI Analysis

Customer Engagement

Use Case Ideation

AI Playbook Development

AI Play Pruning

QA & Testing

Phase 1

Phase 2

Phase 3

Agile AI - Feasibility Exploration

Data Plumbing & Exploration

Use Case Ideation

AI Playbook Development

Phase 1

• Business team is responsible for sourcing problems from across the organization
• Problem statements are best if they can have dollar impact (estimated) assigned to them

• SME's and DS practitioners collaborate to start developing different ways of framing emerging problem statements into optimization language

• Regardless of problem / solution identification, data team should start immediately working to identify potential data sources, data schema and developing necessary data stores and ETL logic
• Initial, lightweight data analysis and exploration can help guide ETL activities as well as feed back up to use case identification

Agile AI - Value Selection

Model Development

ROI Analysis

AI Play Pruning

Phase 2

• Using cost inputs and feasibility assessments from data team, business team should work to prioritize initiatives based on overall ROI

• Feedback from data team should guide aggressive elimination of options
• PM's and SME's should work to update cost estimates for full product realization

• Data team should begin iterating towards initial, local models for first set of use cases to explore
• Models should use sample of data and research is focused on validating assumptions about predictive power against assumed use case

Agile AI - Model Operationalization

Model Scaling & Packaging

Customer Engagement

QA & Testing

Phase 3

• Business team should take emerging solutions / offerings and bring them to customers to guide final investment decisions (MVP and roadmap planning)

• SME's should be collaborating with product realization team to test Alpha / Beta versions against proposed intelligent value statement

• As scope is honed in and non-fruitful research swim lanes are cut, work moves from model development to model realization
• Data science team's focus is on "packaging" up models (exporting trained weights, moving script code into version controlled libraries, etc.)

Running an AI Playbook

Use-Case Driven Discussions

Problem

Framing

Framing

S

S

S

S

Possible Solutions

(Management) Science

(Data) Science

Creative Art

Research Swim Lanes

Framing the Learning Problem

Problem

"We need to improve our search click-through rate by at least 10%"

Framing

Re-frame problem into two parts - content selection and content ranking.

1

2

Content Selection

Content Ranking

• More traditional IR problem
• Focus on recall vs precision
• Ample literature for review

• Varied ML tasks - classification, regression, matrix factorization

Optimizing Research Progress

Minimal Viable Model

Problem

Framing

1

2

Time to Value

Resource Constraints

Hard vs Soft Science

VS

Hard Science

Soft Science

Bag of Words for Sentiment Classification

Entity Coreference Resolution

• Simple implementation / debugging
• Pre-existing libraries
• Known prior art

• Active research / state-of-art (in literature)
• No extensive feature engineering

Example: Customer Service Optimization

The Customer Service Challenge

Customer Service Operations

• Typically an organizations largest source of (non-skilled) labor cost
• Despite advances in automation, majority of consumers still prefer human interaction
• Historical approaches have focused on reducing the variable labor cost (i.e. - outsourcing support)

Re-Framing the Challenge

Problem

"Reduce total customer service cost by at least 25%."

Framing

Cost = Total Minutes x Cost / Minute

1

2

Remove Demand

Improve Rep Support Efficiency

• Identify common customer scenarios and automate

• Pre-qualify customer via questions or CRM
• Develop / iterate methods for exposing to service rep

Minimize Average Time / Rep

3

Pre-Educate Customer

• Add chatbot self-education capabilities during hold

Potential Solutions Emerge

1

2

Remove Demand

Improve Rep Support Efficiency

• Leverage unsupervised NLP methods to group common "problems" together
• Use small taxonomy team to seed a "customer service" ontology
• Match text to labels of ontology

• Conduct interviews with support staff to identify customer info needs
• Research, ETL and connect inbound customer to CRM

3

Pre-Educate Customer

• Identify common "information" needs in support transcripts
• Prototype chatbot interaction (single script)

How to Contact Me

garrett@dataexhaust.io

• Follow up questions
• Model development / implementation
• Product team training
• Moral support

Thank you!