Optimization of Tree Ensembles
Real World Problem
But also want to know the strategy to make some differences (a.k.a. modify the sample)
When applying a classifier,
sometimes we not only just want to know whether a sample belongs to a class
Examples
- Medical attention
- Company decision
First step
Problem Definition
Given us a classifier C(⋅), and an n-features vector X={x1,x2,...,xn}
The object is to find another X′ so that X′=argX′maxC(X′)
How?
Focus on single model
Random Forest
Random Forest
RF is C(X)=t=1∑Tλtft(X) here fi(⋅) is a tree in the forest.
Example from Iris dataset
How to formulate the problem?
idea: consider it to be a MIP problem
Terminology
Let leaves(t) be the set of leaves or terminal nodes of tree t.
Let splits(t) denote the set of splits of tree t (non-terminal nodes).
Let left(s) be the set of leaves that are accessible from the left branch, and same as the right(s)
let V(s)∈{1,...,n} denote the variable that participates in split s,
and let C(s) denote the set of values of variable i that participate in the split query of s.
Object function
x,ymaxt=1∑Tℓ∈leaves(t)∑λt⋅pt,ℓ⋅yt,ℓ
Constraints
- the observation falls in exactly one of the leaves of each tree t
- if 1 observation falls into the a sub-tree, then no observation could fall into the other part of sub-tree
- the indicator y must be in {0, 1}
Intermediate variable
xi,j indicates that if a feature Xi fulfills the predicate of that node, i.e. Xi falls into left branch of the tree
the observation falls in exactly one of the leaves of each tree t
\sum_{\ell \in \textbf{leaves}(t)}y_{t,\ell} = 1, \forall t \in \{1, ..., T\}
∑ℓ∈leaves(t)yt,ℓ=1,∀t∈{1,...,T}
if 1 observation falls into the a sub-tree, then no observation could fall into the other part of sub-tree
\sum_{\ell \in \textbf{left}(s)} y_{t, \ell} \leq \sum_{j \in \textbf{C}(s)}x_{\textbf{V}(s), j}, \ \forall t \in \{1, ..., T\}, s \in \textbf{splits}(t)
∑ℓ∈left(s)yt,ℓ≤∑j∈C(s)xV(s),j, ∀t∈{1,...,T},s∈splits(t)
\sum_{\ell \in \textbf{right}(s)} y_{t, \ell} \leq 1 - \sum_{j \in \textbf{C}(s)} x_{\textbf{V}(s), j}, \forall t \in \{1, ..., T\}, s \in \textbf{splits}(t)
∑ℓ∈right(s)yt,ℓ≤1−∑j∈C(s)xV(s),j,∀t∈{1,...,T},s∈splits(t)
Some additional constraints on x
\sum_{j=1}^{K_i} x_{i,j} = 1, \forall i \in \mathcal{C}
∑j=1Kixi,j=1,∀i∈C
x_{i,j} \leq x_{i, j+1}, \forall i \in \mathcal{N}, j \in \{1, ..., K_i - 1\}
xi,j≤xi,j+1,∀i∈N,j∈{1,...,Ki−1}
x_{i,j} \in \{0, 1\}, \forall i \in \{1, ..., n\}, j \in \{1, ..., K_i - 1\},
xi,j∈{0,1},∀i∈{1,...,n},j∈{1,...,Ki−1},
the indicator y must be in {0, 1}
y_{t, \ell} \geq 0, \forall t \in \{1,...,T\}, \ell \in \textbf{leaves}(t)
yt,ℓ≥0,∀t∈{1,...,T},ℓ∈leaves(t)
Trick kicks in!
All in one
\max_{\textbf{x},\textbf{y}} \sum_{t=1}^{T}\sum_{\ell \in \textbf{leaves}(t)} \lambda_t \cdot p_{t,\ell} \cdot y_{t, \ell}
maxx,y∑t=1T∑ℓ∈leaves(t)λt⋅pt,ℓ⋅yt,ℓ
subject \ to\ \sum_{\ell \in \textbf{leaves}(t)}y_{t,\ell} = 1, \forall t \in \{1, ..., T\}
subject to ∑ℓ∈leaves(t)yt,ℓ=1,∀t∈{1,...,T}
\sum_{\ell \in \textbf{left}(s)} y_{t, \ell} \leq \sum_{j \in \textbf{C}(s)}x_{\textbf{V}(s), j}, \ \forall t \in \{1, ..., T\}, s \in \textbf{splits}(t)
∑ℓ∈left(s)yt,ℓ≤∑j∈C(s)xV(s),j, ∀t∈{1,...,T},s∈splits(t)
\sum_{\ell \in \textbf{right}(s)} y_{t, \ell} \leq 1 - \sum_{j \in \textbf{C}(s)} x_{\textbf{V}(s), j}, \forall t \in \{1, ..., T\}, s \in \textbf{splits}(t)
∑ℓ∈right(s)yt,ℓ≤1−∑j∈C(s)xV(s),j,∀t∈{1,...,T},s∈splits(t)
\sum_{j=1}^{K_i} x_{i,j} = 1, \forall i \in \mathcal{C}
∑j=1Kixi,j=1,∀i∈C
x_{i,j} \leq x_{i, j+1}, \forall i \in \mathcal{N}, j \in \{1, ..., K_i - 1\}
xi,j≤xi,j+1,∀i∈N,j∈{1,...,Ki−1}
x_{i,j} \in \{0, 1\}, \forall i \in \{1, ..., n\}, j \in \{1, ..., K_i - 1\},
xi,j∈{0,1},∀i∈{1,...,n},j∈{1,...,Ki−1},
y_{t, \ell} \geq 0, \forall t \in \{1,...,T\}, \ell \in \textbf{leaves}(t)
yt,ℓ≥0,∀t∈{1,...,T},ℓ∈leaves(t)
Approximation
It's quite time-consuming to solve the original problem
\sum_{\ell \in \textbf{left}(s)} y_{t, \ell} \leq \sum_{j \in \textbf{C}(s)}x_{\textbf{V}(s), j}, \ \forall t \in \{1, ..., T\}, s \in \textbf{splits}(t)
∑ℓ∈left(s)yt,ℓ≤∑j∈C(s)xV(s),j, ∀t∈{1,...,T},s∈splits(t)
\sum_{\ell \in \textbf{right}(s)} y_{t, \ell} \leq 1 - \sum_{j \in \textbf{C}(s)} x_{\textbf{V}(s), j}, \forall t \in \{1, ..., T\}, s \in \textbf{splits}(t)
∑ℓ∈right(s)yt,ℓ≤1−∑j∈C(s)xV(s),j,∀t∈{1,...,T},s∈splits(t)
Traverse the whole forest, O(k*2^n)!
Idea: what if we do not search to the deepest of the tree?
Ω={(t,s)∣t∈{1,...,T},s∈splits(t)}
First define
\sum_{\ell \in \textbf{left}(s)} y_{t, \ell} \leq \sum_{j \in \textbf{C}(s)}x_{\textbf{V}(s), j}, \ \forall (t,s) \in \bar{\Omega}
∑ℓ∈left(s)yt,ℓ≤∑j∈C(s)xV(s),j, ∀(t,s)∈Ωˉ
\sum_{\ell \in \textbf{right}(s)} y_{t, \ell} \leq 1 - \sum_{j \in \textbf{C}(s)} x_{\textbf{V}(s), j}, \forall (t,s) \in \bar{\Omega}
∑ℓ∈right(s)yt,ℓ≤1−∑j∈C(s)xV(s),j,∀(t,s)∈Ωˉ
\sum_{\ell \in \textbf{left}(s)} y_{t, \ell} \leq \sum_{j \in \textbf{C}(s)}x_{\textbf{V}(s), j}, \ \forall t \in \{1, ..., T\}, s \in \textbf{splits}(t)
∑ℓ∈left(s)yt,ℓ≤∑j∈C(s)xV(s),j, ∀t∈{1,...,T},s∈splits(t)
\sum_{\ell \in \textbf{right}(s)} y_{t, \ell} \leq 1 - \sum_{j \in \textbf{C}(s)} x_{\textbf{V}(s), j}, \forall t \in \{1, ..., T\}, s \in \textbf{splits}(t)
∑ℓ∈right(s)yt,ℓ≤1−∑j∈C(s)xV(s),j,∀t∈{1,...,T},s∈splits(t)
Proposion
Z^*_{MIO,1} \geq Z^*_{MIO,2} \geq... \geq Z^*_{MIO,d_{max}} \geq Z^*_{MIO}
ZMIO,1∗≥ZMIO,2∗≥...≥ZMIO,dmax∗≥ZMIO∗
Where Z is the objective value
Theorem
\delta_{t,s} = max \{
\max_{\ell \in left(s)} p_{t,\ell} - \min_{\ell \in left(s)} p_{t,\ell}, \max_{\ell \in right(s)} p_{t,\ell} - \min_{\ell \in right(s)} p_{t,\ell} \}
δt,s=max{maxℓ∈left(s)pt,ℓ−minℓ∈left(s)pt,ℓ,maxℓ∈right(s)pt,ℓ−minℓ∈right(s)pt,ℓ}
\Delta_t = \max_{s \in splict(t,d)} \delta_{t,s}
Δt=maxs∈splict(t,d)δt,s
Z^*_{MIO,d} - \sum^T_{t=1} \lambda_t \Delta_t \leq Z_d \leq Z^*_{MIO} \leq Z^*_{MIO,d}
ZMIO,d∗−∑t=1TλtΔt≤Zd≤ZMIO∗≤ZMIO,d∗
Experiments
Experiments
Experiments
Optimization of Tree Ensembles