Enabling SQL-ML Explanation to Debug

Training Data

Weiyuan Wu

youngw@sfu.ca

Committee:

Dr. Jiannan Wang - Senior Supervisor

Dr. Jian Pei - Supervisor

Dr. Oliver Schulte - Examiner

Dr. Steven Bergner - Chair

Thesis Defense

Aug. 22. 2019

SQL-ML Query

SQL-ML System

Why SQL-ML System

• Democratize AI
• Improve Productivity
• Facilitate Management

Training Data is Often Corrupted

Corrupted training data

The Need for SQL-ML Explanation

A Debugging Workflow

• Software debugging: 

Existing Approches

• SQL Explanation
  • Why Not? [A Chapman et al. SIGMOD'09]
  • Tiresias [A Meliou et al. SIGMOD'12]
  • Scorpion [E Wu et al. VLDB'13]
• ML Explanation
  • Influence Function [PW Koh et al. ICML'17]
  • DUTI [X Zhang et al. AAAI'18]

SQL & ML Explanation

Simple Combination

• SQL explanation: produces labels for querying set
• ML explanation: requires labels for querying set to debug
• HowToComp: 

SQL Exp.

ML Exp.

Querying Set Points

Possible Corruptions

Ambiguity

SQL Exp.

ML Exp.

Querying Set Points

Possible Corruptions

• HowToComp: Multiple ways to select querying set points. 

Agenda

• Motivation & Background
• Problem Definition & Challenges
• Our Solution
• Experiments
• Conclusions

The SQL-ML Explanation Problem

Inputs:

• A training dataset
• A SQL-ML query
• A complaint (e.g. count is too high)

Output:

• Possible corrupted training data points

Challenges

• Problem Formulation
  • Holistic
  • Discrete + Continous Optimization
• Efficient Algorithm
  • Need to train \(n^k\) models

Agenda

• Motivation & Background
• Problem Definition & Challenges
• Our Solution
• Experiments
• Conclusions

InfComp: Influence & Complaint

• Compute influence on the differentiable query result
  1. Make SQL query result differentiable w.r.t. model parameters by provenance polynomial
  2. Training set debug using Influence Function

Differentiable Query Result

SQL Provenance

• SQL operators are isomorphic to some semimodules
• \(\pi, \sigma, \gamma, \bowtie, \cup\) can be expressed using a boolean semimodule or integer semimodule, depending on the set or bag semantics
• Example:

Connect ML and SQL

• Issue: Non-differentiable
  • ​\(f(P1,P2,P3) = P1 + P2 + P3\)​​
  • \(P1,P2,P3 \in \{0,1\}\)
• Solution: Relaxation
  • Relax binaries into continuous: \(P1',P2',P3' \in [0,1]\), thus \(f(P1',P2',P3') \in [0,3]\)​
  • Replace \(P1', P2', P3'\) with the probabilistic output from the model

• Now the query output \(\mathcal{Q}\) is differentiable w.r.t. ML params!

Training Set Debugging

Training data

Count(*)

......

Influence Function

Where:

\(\theta^* = \argmin_{\theta} L(\theta)\)

\(L(\theta) = \sum_{(x,y) \in T} \ell( f(x, \theta),y) \)

\(\theta_{\epsilon}^* = \argmin_{\theta} L(\theta) + \epsilon \cdot \ell( f(x', \theta),y')\)

\(H_{\theta} = \nabla_{\theta}^2 L(\theta) \)

• Influence Function: \[\left. \frac{d \theta_{\epsilon}^*}{d\epsilon}\right|_{\epsilon=0} = \lim_{\epsilon \to 0} \frac{\theta_{\epsilon}^* - \theta^*}{\epsilon}  = - H_{\theta^*}^{-1} \nabla_{\theta^*} \ell( f(x', \theta^*),y') \]
• InfComp: ​\[ \left. \frac{d q(\theta_{\epsilon}^*) }{d\epsilon}\right|_{{\epsilon=0}} = - \nabla_{\theta} q(\theta^*) \left. \frac{d \theta_{\epsilon}^*}{d\epsilon}\right|_{\epsilon=0} = - \nabla_{\theta} q(\theta^*) \ H_{\theta^*}^{-1} \ \nabla_{\theta} \ell( f(x', \theta^*),y') \]

InfComp Algorithm

K most possible corruptions

Agenda

• Motivation & Background
• Problem Definition & Challenges
• Our Solution
• Experiments
• Conclusions

Experiment: Compared Methods

• Baselines:
  • Complaint agnostic:
    • Loss: delete training points with maximum loss values
    • InfLoss\(^1\): delete training points with maximum self-influences
  • Complaint aware:
    • HowToComp\(^{1,2}\): delete training points by first using SQL explanation tools and then Influence Function
• Our approach:
  • InfComp: Complaint aware, deleting training points by holistically considering SQL and ML

Experiment: Tasks

• Entity resolution task: DBLP-GOOG dataset

• Image recognition task: MNIST dataset

• Spam classification task: ENRON dataset

Corruptions

• Random Label Flip: DBLP/MNIST
• Positive/Negative Label Flip: DBLP/MNIST
• Label Generation from a corrupted labeler: MNIST
• Label Flip by word: ENRON

Experiment Results: MNIST

Experiment Results: DBLP & ENRON

The Top K Corrutption Recall on DBLP. The closer to the ground truth line, the better.

Median Corruption

High Corruption

The Top K Corrutption Recall on ENRON. The closer to the ground truth line, the better.

"deal" Corruption

"http" Corruption

Experiment Result: Time Overhead

Average time for finding one possible corruption, the lower the better. Loss: 1x, InfLoss: 2.4x, InfComp: 1.3x, HowToComp: 2x.

Conclusions

1. To our best knowledge, InfComp is the first to debug SQL-ML query.
2. Infcomp can leverage the complaint to find more corrupted training points compared to the baselines. 
3. InfComp is fast and adds little time overhead. 

Future Works

• Richer set of explanations
• Integration to analytical databases
• Why not complaints

Thank you!