Enduring principles:

1. Chop up signal into patches (divide and conquer)
2. Process each patch identically (and in parallel)

Lessons from CNNs

CNN

• Importantly, all these learned projection weights \(W\) are shared along the token sequence.
• Same "operation" repeated.

Transformers

Interpretability

Outline

• Recap (transforermers)
• Non-parametric models
  • interpretability
  • ease of use/simplicity
• Decision tree
  • Terminologies
  • Learn via the BuildTree algorithm
    • Regression
    • Classification 
• ​Nearest neighbor

• does not mean "no parameters"
• there are still parameters to be learned to build a hypothesis/model.
• just that, the model/hypothesis does not have a fixed parameterization. 
• (e.g. even the number of parameters can change.)

Non-parametric models 

• Decision trees and 
• Nearest neighbor

are the classical examples of non-parametric models

Root node

Internal (decision) node

Leaf (terminal) node

Split dimension

Split value

features:

• \(x_1\): temperature (deg C)
• \(x_2\): precipitation (cm/hr)

labels:
\(y\): km run

Tree defines an axis-aligned “partition” of the feature space:

• input \(I\): set of indices
• \(k\): hyper-parameter, maximum leaf "size", i.e. how many training data ended in that leaf node.
• \(\hat y\): (intermediate) prediction

\(\operatorname{BuildTree}(I;k)\)

1. if \(|I| > k\)
2.       for each split dim \(j\) and split value \(s\)
3.             Set   \(I_{j, s}^{+}=\left\{i \in I \mid x_j^{(i)} \geq s\right\}\)
4.             Set   \(I_{j, s}^{-}=\left\{i \in I \mid x_j^{(i)}<s\right\}\)
5.             Set   \(\hat{y}_{j, s}^{+}=\)average \(_{i \in I_{j, s}^{+}}  y^{(i)}\)
6.             Set   \(\hat{y}_{j, s}^{-}=\)average \(_{i \in I_{j, s}^{-}}  y^{(i)}\)
7.             Set   \(E_{j, s}=\sum_{i \in I_{j, s}^{+}}\left(y^{(i)}-\hat{y}_{j, s}^{+}\right)^2+\sum_{i \in I_{j, s}^{-}}\left(y^{(i)}-\hat{y}_{j, s}^{-}\right)^2\)
8.       Set   \(\left(j^*, s^*\right)=\arg \min _{j, s} E_{j, s}\)
9. else
10.       Set   \(\hat{y}=\) average \(_{i \in I}  y^{(i)}\)
11.       return \(\operatorname{LEAF}\)(leave_value=\(\hat{y})\)
12. return \(\operatorname{Node}\left(j^*, s^*, \operatorname{BuildTree}\left(I_{j^*, s^*}^{-}, k\right), \operatorname{BuildTree}\left(I_{j^*, s^*}^{+}, k\right)\right)\)

• \(j\): split dimension
• \(s\): split value

\(\operatorname{BuildTree}(I;k)\)

1. if \(|I| > k\)
2.       for each split dim \(j\) and split value \(s\)
3.             Set   \(I_{j, s}^{+}=\left\{i \in I \mid x_j^{(i)} \geq s\right\}\)
4.             Set   \(I_{j, s}^{-}=\left\{i \in I \mid x_j^{(i)}<s\right\}\)
5.             Set   \(\hat{y}_{j, s}^{+}=\)average \(_{i \in I_{j, s}^{+}}  y^{(i)}\)
6.             Set   \(\hat{y}_{j, s}^{-}=\)average \(_{i \in I_{j, s}^{-}}  y^{(i)}\)
7.             Set   \(E_{j, s}=\sum_{i \in I_{j, s}^{+}}\left(y^{(i)}-\hat{y}_{j, s}^{+}\right)^2+\sum_{i \in I_{j, s}^{-}}\left(y^{(i)}-\hat{y}_{j, s}^{-}\right)^2\)
8.       Set   \(\left(j^*, s^*\right)=\arg \min _{j, s} E_{j, s}\)
9. else
10.       Set   \(\hat{y}=\) average \(_{i \in I}  y^{(i)}\)
11.       return \(\operatorname{LEAF}\)(leave_value=\(\hat{y})\)
12. return \(\operatorname{Node}\left(j^*, s^*, \operatorname{BuildTree}\left(I_{j^*, s^*}^{-}, k\right), \operatorname{BuildTree}\left(I_{j^*, s^*}^{+}, k\right)\right)\)

• Choose \(k=2\)
• \(\operatorname{BuildTree}(\{1,2,3\};2)\)
• Line 1 true
• Consider a fixed \((j, s)\)
  • ​\(I_{j, s}^{+} = \{2,3\}\)
  • \(I_{j, s}^{-} = \{1\}\)
  • \(\hat{y}_{j, s}^{+} = 5\)
  • \(\hat{y}_{j, s}^{-} = 0\)
  • \(E_{j, s} =0\)

\(\operatorname{BuildTree}(I;k)\)

1. if \(|I| > k\)
2.       for each split dim \(j\) and split value \(s\)
3.             Set   \(I_{j, s}^{+}=\left\{i \in I \mid x_j^{(i)} \geq s\right\}\)
4.             Set   \(I_{j, s}^{-}=\left\{i \in I \mid x_j^{(i)}<s\right\}\)
5.             Set   \(\hat{y}_{j, s}^{+}=\)average \(_{i \in I_{j, s}^{+}}  y^{(i)}\)
6.             Set   \(\hat{y}_{j, s}^{-}=\)average \(_{i \in I_{j, s}^{-}}  y^{(i)}\)
7.             Set   \(E_{j, s}=\sum_{i \in I_{j, s}^{+}}\left(y^{(i)}-\hat{y}_{j, s}^{+}\right)^2+\sum_{i \in I_{j, s}^{-}}\left(y^{(i)}-\hat{y}_{j, s}^{-}\right)^2\)
8.       Set   \(\left(j^*, s^*\right)=\arg \min _{j, s} E_{j, s}\)
9. else
10.       Set   \(\hat{y}=\) average \(_{i \in I}  y^{(i)}\)
11.       return \(\operatorname{LEAF}\)(leave_value=\(\hat{y})\)
12. return \(\operatorname{Node}\left(j^*, s^*, \operatorname{BuildTree}\left(I_{j^*, s^*}^{-}, k\right), \operatorname{BuildTree}\left(I_{j^*, s^*}^{+}, k\right)\right)\)

• Choose \(k=2\)
• \(\operatorname{BuildTree}(\{1,2,3\};2)\)
• Line 1 true
• Consider a fixed \((j, s)\)
  • ​\(I_{j, s}^{+} = \{2,3\}\)
  • \(I_{j, s}^{-} = \{1\}\)
  • \(\hat{y}_{j, s}^{+} = 5\)
  • \(\hat{y}_{j, s}^{-} = 0\)
  • \(E_{j, s} =0\)

\(\operatorname{BuildTree}(I;k)\)

1. if \(|I| > k\)
2.       for each split dim \(j\) and split value \(s\)
3.             Set   \(I_{j, s}^{+}=\left\{i \in I \mid x_j^{(i)} \geq s\right\}\)
4.             Set   \(I_{j, s}^{-}=\left\{i \in I \mid x_j^{(i)}<s\right\}\)
5.             Set   \(\hat{y}_{j, s}^{+}=\)average \(_{i \in I_{j, s}^{+}}  y^{(i)}\)
6.             Set   \(\hat{y}_{j, s}^{-}=\)average \(_{i \in I_{j, s}^{-}}  y^{(i)}\)
7.             Set   \(E_{j, s}=\sum_{i \in I_{j, s}^{+}}\left(y^{(i)}-\hat{y}_{j, s}^{+}\right)^2+\sum_{i \in I_{j, s}^{-}}\left(y^{(i)}-\hat{y}_{j, s}^{-}\right)^2\)
8.       Set   \(\left(j^*, s^*\right)=\arg \min _{j, s} E_{j, s}\)
9. else
10.       Set   \(\hat{y}=\) average \(_{i \in I}  y^{(i)}\)
11.       return \(\operatorname{LEAF}\)(leave_value=\(\hat{y})\)
12. return \(\operatorname{Node}\left(j^*, s^*, \operatorname{BuildTree}\left(I_{j^*, s^*}^{-}, k\right), \operatorname{BuildTree}\left(I_{j^*, s^*}^{+}, k\right)\right)\)

• So for line 2: a finite number of \((j, s)\) combo suffices (those splits in-between data points)
• Line 8 picks the "best" among these finite combos. (random tie-breaking)

\(\operatorname{BuildTree}(I;k)\)

1. if \(|I| > k\)
2.       for each split dim \(j\) and split value \(s\)
3.             Set   \(I_{j, s}^{+}=\left\{i \in I \mid x_j^{(i)} \geq s\right\}\)
4.             Set   \(I_{j, s}^{-}=\left\{i \in I \mid x_j^{(i)}<s\right\}\)
5.             Set   \(\hat{y}_{j, s}^{+}=\)average \(_{i \in I_{j, s}^{+}}  y^{(i)}\)
6.             Set   \(\hat{y}_{j, s}^{-}=\)average \(_{i \in I_{j, s}^{-}}  y^{(i)}\)
7.             Set   \(E_{j, s}=\sum_{i \in I_{j, s}^{+}}\left(y^{(i)}-\hat{y}_{j, s}^{+}\right)^2+\sum_{i \in I_{j, s}^{-}}\left(y^{(i)}-\hat{y}_{j, s}^{-}\right)^2\)
8.       Set   \(\left(j^*, s^*\right)=\arg \min _{j, s} E_{j, s}\)
9. else
10.       Set   \(\hat{y}=\) average \(_{i \in I}  y^{(i)}\)
11.       return \(\operatorname{LEAF}\)(leave_value=\(\hat{y})\)
12. return \(\operatorname{Node}\left(j^*, s^*, \operatorname{BuildTree}\left(I_{j^*, s^*}^{-}, k\right), \operatorname{BuildTree}\left(I_{j^*, s^*}^{+}, k\right)\right)\)

Suppose line 8 sets this \((j^*,s^*) = (1, 1.7)\)

then 12 recursion

\(\operatorname{BuildTree}(I;k)\)

1. if \(|I| > k\)
2.       for each split dim \(j\) and split value \(s\)
3.             Set   \(I_{j, s}^{+}=\left\{i \in I \mid x_j^{(i)} \geq s\right\}\)
4.             Set   \(I_{j, s}^{-}=\left\{i \in I \mid x_j^{(i)}<s\right\}\)
5.             Set   \(\hat{y}_{j, s}^{+}=\)average \(_{i \in I_{j, s}^{+}}  y^{(i)}\)
6.             Set   \(\hat{y}_{j, s}^{-}=\)average \(_{i \in I_{j, s}^{-}}  y^{(i)}\)
7.             Set   \(E_{j, s}=\sum_{i \in I_{j, s}^{+}}\left(y^{(i)}-\hat{y}_{j, s}^{+}\right)^2+\sum_{i \in I_{j, s}^{-}}\left(y^{(i)}-\hat{y}_{j, s}^{-}\right)^2\)
8.       Set   \(\left(j^*, s^*\right)=\arg \min _{j, s} E_{j, s}\)
9. else
10.       Set   \(\hat{y}=\) average \(_{i \in I}  y^{(i)}\)
11.       return \(\operatorname{LEAF}\)(leave_value=\(\hat{y})\)
12. return \(\operatorname{Node}\left(j^*, s^*, \operatorname{BuildTree}\left(I_{j^*, s^*}^{-}, k\right), \operatorname{BuildTree}\left(I_{j^*, s^*}^{+}, k\right)\right)\)

Line 8 sets this \((j^*,s^*)\)

Line 12 recursion

\(\operatorname{BuildTree}(I;k)\)

1. if \(|I| > k\)
2.       for each split dim \(j\) and split value \(s\)
3.             Set   \(I_{j, s}^{+}=\left\{i \in I \mid x_j^{(i)} \geq s\right\}\)
4.             Set   \(I_{j, s}^{-}=\left\{i \in I \mid x_j^{(i)}<s\right\}\)
5.             Set   \(\hat{y}_{j, s}^{+}=\)average \(_{i \in I_{j, s}^{+}}  y^{(i)}\)
6.             Set   \(\hat{y}_{j, s}^{-}=\)average \(_{i \in I_{j, s}^{-}}  y^{(i)}\)
7.             Set   \(E_{j, s}=\sum_{i \in I_{j, s}^{+}}\left(y^{(i)}-\hat{y}_{j, s}^{+}\right)^2+\sum_{i \in I_{j, s}^{-}}\left(y^{(i)}-\hat{y}_{j, s}^{-}\right)^2\)
8.       Set   \(\left(j^*, s^*\right)=\arg \min _{j, s} E_{j, s}\)
9. else
10.       Set   \(\hat{y}=\) average \(_{i \in I}  y^{(i)}\)
11.       return \(\operatorname{LEAF}\)(leave_value=\(\hat{y})\)
12. return \(\operatorname{Node}\left(j^*, s^*, \operatorname{BuildTree}\left(I_{j^*, s^*}^{-}, k\right), \operatorname{BuildTree}\left(I_{j^*, s^*}^{+}, k\right)\right)\)

Line 8 sets this \((j^*,s^*)\)

Line 12 recursion

\(\operatorname{BuildTree}(I;k)\)

1. if \(|I| > k\)
2.       for each split dim \(j\) and split value \(s\)
3.             Set   \(I_{j, s}^{+}=\left\{i \in I \mid x_j^{(i)} \geq s\right\}\)
4.             Set   \(I_{j, s}^{-}=\left\{i \in I \mid x_j^{(i)}<s\right\}\)
5.             Set   \(\hat{y}_{j, s}^{+}=\)average \(_{i \in I_{j, s}^{+}}  y^{(i)}\)
6.             Set   \(\hat{y}_{j, s}^{-}=\)average \(_{i \in I_{j, s}^{-}}  y^{(i)}\)
7.             Set   \(E_{j, s}=\sum_{i \in I_{j, s}^{+}}\left(y^{(i)}-\hat{y}_{j, s}^{+}\right)^2+\sum_{i \in I_{j, s}^{-}}\left(y^{(i)}-\hat{y}_{j, s}^{-}\right)^2\)
8.       Set   \(\left(j^*, s^*\right)=\arg \min _{j, s} E_{j, s}\)
9. else
10.       Set   \(\hat{y}=\) average \(_{i \in I}  y^{(i)}\)
11.       return \(\operatorname{LEAF}\)(leave_value=\(\hat{y})\)
12. return \(\operatorname{Node}\left(j^*, s^*, \operatorname{BuildTree}\left(I_{j^*, s^*}^{-}, k\right), \operatorname{BuildTree}\left(I_{j^*, s^*}^{+}, k\right)\right)\)

Line 8 sets this \((j^*,s^*)\)

Line 12 recursion

\(\operatorname{BuildTree}(I;k)\)

1. if \(|I| > k\)
2.       for each split dim \(j\) and split value \(s\)
3.             Set   \(I_{j, s}^{+}=\left\{i \in I \mid x_j^{(i)} \geq s\right\}\)
4.             Set   \(I_{j, s}^{-}=\left\{i \in I \mid x_j^{(i)}<s\right\}\)
5.             Set   \(\hat{y}_{j, s}^{+}=\)average \(_{i \in I_{j, s}^{+}}  y^{(i)}\)
6.             Set   \(\hat{y}_{j, s}^{-}=\)average \(_{i \in I_{j, s}^{-}}  y^{(i)}\)
7.             Set   \(E_{j, s}=\sum_{i \in I_{j, s}^{+}}\left(y^{(i)}-\hat{y}_{j, s}^{+}\right)^2+\sum_{i \in I_{j, s}^{-}}\left(y^{(i)}-\hat{y}_{j, s}^{-}\right)^2\)
8.       Set   \(\left(j^*, s^*\right)=\arg \min _{j, s} E_{j, s}\)
9. else
10.       Set   \(\hat{y}=\) average \(_{i \in I}  y^{(i)}\)
11.       return \(\operatorname{LEAF}\)(leave_value=\(\hat{y})\)
12. return \(\operatorname{Node}\left(j^*, s^*, \operatorname{BuildTree}\left(I_{j^*, s^*}^{-}, k\right), \operatorname{BuildTree}\left(I_{j^*, s^*}^{+}, k\right)\right)\)

Line 8 sets this \((j^*,s^*)\)

Line 12 recursion

\(\operatorname{BuildTree}(I;k)\)

1. if \(|I| > k\)
2.       for each split dim \(j\) and split value \(s\)
3.             Set   \(I_{j, s}^{+}=\left\{i \in I \mid x_j^{(i)} \geq s\right\}\)
4.             Set   \(I_{j, s}^{-}=\left\{i \in I \mid x_j^{(i)}<s\right\}\)
5.             Set   \(\hat{y}_{j, s}^{+}=\)average \(_{i \in I_{j, s}^{+}}  y^{(i)}\)
6.             Set   \(\hat{y}_{j, s}^{-}=\)average \(_{i \in I_{j, s}^{-}}  y^{(i)}\)
7.             Set   \(E_{j, s}=\sum_{i \in I_{j, s}^{+}}\left(y^{(i)}-\hat{y}_{j, s}^{+}\right)^2+\sum_{i \in I_{j, s}^{-}}\left(y^{(i)}-\hat{y}_{j, s}^{-}\right)^2\)
8.       Set   \(\left(j^*, s^*\right)=\arg \min _{j, s} E_{j, s}\)
9. else
10.       Set   \(\hat{y}=\) average \(_{i \in I}  y^{(i)}\)
11.       return \(\operatorname{LEAF}\)(leave_value=\(\hat{y})\)
12. return \(\operatorname{Node}\left(j^*, s^*, \operatorname{BuildTree}\left(I_{j^*, s^*}^{-}, k\right), \operatorname{BuildTree}\left(I_{j^*, s^*}^{+}, k\right)\right)\)

\(\operatorname{BuildTree}(I;k)\)

1. if \(|I| > k\)
2.       for each split dim \(j\) and split value \(s\)
3.             Set   \(I_{j, s}^{+}=\left\{i \in I \mid x_j^{(i)} \geq s\right\}\)
4.             Set   \(I_{j, s}^{-}=\left\{i \in I \mid x_j^{(i)}<s\right\}\)
5.             Set   \(\hat{y}_{j, s}^{+}=\)average \(_{i \in I_{j, s}^{+}}  y^{(i)}\)
6.             Set   \(\hat{y}_{j, s}^{-}=\)average \(_{i \in I_{j, s}^{-}}  y^{(i)}\)
7.             Set   \(E_{j, s}=\sum_{i \in I_{j, s}^{+}}\left(y^{(i)}-\hat{y}_{j, s}^{+}\right)^2+\sum_{i \in I_{j, s}^{-}}\left(y^{(i)}-\hat{y}_{j, s}^{-}\right)^2\)
8.       Set   \(\left(j^*, s^*\right)=\arg \min _{j, s} E_{j, s}\)
9. else
10.       Set   \(\hat{y}=\) average \(_{i \in I}  y^{(i)}\)
11.       return \(\operatorname{LEAF}\)(leave_value=\(\hat{y})\)
12. return \(\operatorname{Node}\left(j^*, s^*, \operatorname{BuildTree}\left(I_{j^*, s^*}^{-}, k\right), \operatorname{BuildTree}\left(I_{j^*, s^*}^{+}, k\right)\right)\)

\(\operatorname{BuildTree}(I;k)\)

1. if \(|I| > k\)
2.       for each split dim \(j\) and split value \(s\)
3.             Set   \(I_{j, s}^{+}=\left\{i \in I \mid x_j^{(i)} \geq s\right\}\)
4.             Set   \(I_{j, s}^{-}=\left\{i \in I \mid x_j^{(i)}<s\right\}\)
5.             Set.  \(\hat{y}_{j, s}^{+}=\)average \(_{i \in I_{j, s}^{+}}  y^{(i)}\)
6.             Set   \(\hat{y}_{j, s}^{-}=\)average \(_{i \in I_{j, s}^{-}}  y^{(i)}\)
7.             Set   \(E_{j, s}=\sum_{i \in I_{j, s}^{+}}\left(y^{(i)}-\hat{y}_{j, s}^{+}\right)^2+\sum_{i \in I_{j, s}^{-}}\left(y^{(i)}-\hat{y}_{j, s}^{-}\right)^2\)
8.       Set   \(\left(j^*, s^*\right)=\arg \min _{j, s} E_{j, s}\)
9. else
10.       Set   \(\hat{y}=\) average \(_{i \in I}  y^{(i)}\)
11.       return \(\operatorname{LEAF}\)(leave_value=\(\hat{y})\)
12. return \(\operatorname{Node}\left(j^*, s^*, \operatorname{BuildTree}\left(I_{j^*, s^*}^{-}, k\right), \operatorname{BuildTree}\left(I_{j^*, s^*}^{+}, k\right)\right)\)

\(\operatorname{BuildTree}(I;k)\)

1. if \(|I| > k\)
2.       for each split dim \(j\) and split value \(s\)
3.             Set   \(I_{j, s}^{+}=\left\{i \in I \mid x_j^{(i)} \geq s\right\}\)
4.             Set   \(I_{j, s}^{-}=\left\{i \in I \mid x_j^{(i)}<s\right\}\)
5.             Set.  \(\hat{y}_{j, s}^{+}=\)average \(_{i \in I_{j, s}^{+}}  y^{(i)}\)
6.             Set   \(\hat{y}_{j, s}^{-}=\)average \(_{i \in I_{j, s}^{-}}  y^{(i)}\)
7.             Set   \(E_{j, s}=\sum_{i \in I_{j, s}^{+}}\left(y^{(i)}-\hat{y}_{j, s}^{+}\right)^2+\sum_{i \in I_{j, s}^{-}}\left(y^{(i)}-\hat{y}_{j, s}^{-}\right)^2\)
8.       Set   \(\left(j^*, s^*\right)=\arg \min _{j, s} E_{j, s}\)
9. else
10.       Set   \(\hat{y}=\) average \(_{i \in I}  y^{(i)}\)
11.       return \(\operatorname{LEAF}\)(leave_value=\(\hat{y})\)
12. return \(\operatorname{Node}\left(j^*, s^*, \operatorname{BuildTree}\left(I_{j^*, s^*}^{-}, k\right), \operatorname{BuildTree}\left(I_{j^*, s^*}^{+}, k\right)\right)\)

\(\operatorname{BuildTree}(I;k)\)

1. if \(|I| > k\)
2.       for each split dim \(j\) and split value \(s\)
3.             Set   \(I_{j, s}^{+}=\left\{i \in I \mid x_j^{(i)} \geq s\right\}\)
4.             Set   \(I_{j, s}^{-}=\left\{i \in I \mid x_j^{(i)}<s\right\}\)
5.             Set.  \(\hat{y}_{j, s}^{+}=\)average \(_{i \in I_{j, s}^{+}}  y^{(i)}\)
6.             Set   \(\hat{y}_{j, s}^{-}=\)average \(_{i \in I_{j, s}^{-}}  y^{(i)}\)
7.             Set   \(E_{j, s}=\sum_{i \in I_{j, s}^{+}}\left(y^{(i)}-\hat{y}_{j, s}^{+}\right)^2+\sum_{i \in I_{j, s}^{-}}\left(y^{(i)}-\hat{y}_{j, s}^{-}\right)^2\)
8.       Set   \(\left(j^*, s^*\right)=\arg \min _{j, s} E_{j, s}\)
9. else
10.       Set   \(\hat{y}=\) average \(_{i \in I}  y^{(i)}\)
11.       return \(\operatorname{LEAF}\)(leave_value=\(\hat{y})\)
12. return \(\operatorname{Node}\left(j^*, s^*, \operatorname{BuildTree}\left(I_{j^*, s^*}^{-}, k\right), \operatorname{BuildTree}\left(I_{j^*, s^*}^{+}, k\right)\right)\)

\(\operatorname{BuildTree}(I;k)\)

1. if \(|I| > k\)
2.       for each split dim \(j\) and split value \(s\)
3.             Set   \(I_{j, s}^{+}=\left\{i \in I \mid x_j^{(i)} \geq s\right\}\)
4.             Set   \(I_{j, s}^{-}=\left\{i \in I \mid x_j^{(i)}<s\right\}\)
5.             Set.  \(\hat{y}_{j, s}^{+}=\)average \(_{i \in I_{j, s}^{+}}  y^{(i)}\)
6.             Set   \(\hat{y}_{j, s}^{-}=\)average \(_{i \in I_{j, s}^{-}}  y^{(i)}\)
7.             Set   \(E_{j, s}=\sum_{i \in I_{j, s}^{+}}\left(y^{(i)}-\hat{y}_{j, s}^{+}\right)^2+\sum_{i \in I_{j, s}^{-}}\left(y^{(i)}-\hat{y}_{j, s}^{-}\right)^2\)
8.       Set   \(\left(j^*, s^*\right)=\arg \min _{j, s} E_{j, s}\)
9. else
10.       Set   \(\hat{y}=\) average \(_{i \in I}  y^{(i)}\)
11.       return \(\operatorname{LEAF}\)(leave_value=\(\hat{y})\)
12. return \(\operatorname{Node}\left(j^*, s^*, \operatorname{BuildTree}\left(I_{j^*, s^*}^{-}, k\right), \operatorname{BuildTree}\left(I_{j^*, s^*}^{+}, k\right)\right)\)

\(\operatorname{BuildTree}(I;k)\)

1. if \(|I| > k\)
2.       for each split dim \(j\) and split value \(s\)
3.             Set   \(I_{j, s}^{+}=\left\{i \in I \mid x_j^{(i)} \geq s\right\}\)
4.             Set   \(I_{j, s}^{-}=\left\{i \in I \mid x_j^{(i)}<s\right\}\)
5.             Set.  \(\hat{y}_{j, s}^{+}=\)average \(_{i \in I_{j, s}^{+}}  y^{(i)}\)
6.             Set   \(\hat{y}_{j, s}^{-}=\)average \(_{i \in I_{j, s}^{-}}  y^{(i)}\)
7.             Set   \(E_{j, s}=\sum_{i \in I_{j, s}^{+}}\left(y^{(i)}-\hat{y}_{j, s}^{+}\right)^2+\sum_{i \in I_{j, s}^{-}}\left(y^{(i)}-\hat{y}_{j, s}^{-}\right)^2\)
8.       Set   \(\left(j^*, s^*\right)=\arg \min _{j, s} E_{j, s}\)
9. else
10.       Set   \(\hat{y}=\) average \(_{i \in I}  y^{(i)}\)
11.       return \(\operatorname{LEAF}\)(leave_value=\(\hat{y})\)
12. return \(\operatorname{Node}\left(j^*, s^*, \operatorname{BuildTree}\left(I_{j^*, s^*}^{-}, k\right), \operatorname{BuildTree}\left(I_{j^*, s^*}^{+}, k\right)\right)\)

\(\operatorname{BuildTree}(I;k)\)

1. if \(|I| > k\)
2.       for each split dim \(j\) and split value \(s\)
3.             Set   \(I_{j, s}^{+}=\left\{i \in I \mid x_j^{(i)} \geq s\right\}\)
4.             Set   \(I_{j, s}^{-}=\left\{i \in I \mid x_j^{(i)}<s\right\}\)
5.             Set.  \(\hat{y}_{j, s}^{+}=\)average \(_{i \in I_{j, s}^{+}}  y^{(i)}\)
6.             Set   \(\hat{y}_{j, s}^{-}=\)average \(_{i \in I_{j, s}^{-}}  y^{(i)}\)
7.             Set   \(E_{j, s}=\sum_{i \in I_{j, s}^{+}}\left(y^{(i)}-\hat{y}_{j, s}^{+}\right)^2+\sum_{i \in I_{j, s}^{-}}\left(y^{(i)}-\hat{y}_{j, s}^{-}\right)^2\)
8.       Set   \(\left(j^*, s^*\right)=\arg \min _{j, s} E_{j, s}\)
9. else
10.       Set   \(\hat{y}=\) average \(_{i \in I}  y^{(i)}\)
11.       return \(\operatorname{LEAF}\)(leave_value=\(\hat{y})\)
12. return \(\operatorname{Node}\left(j^*, s^*, \operatorname{BuildTree}\left(I_{j^*, s^*}^{-}, k\right), \operatorname{BuildTree}\left(I_{j^*, s^*}^{+}, k\right)\right)\)

\(\operatorname{BuildTree}(I;k)\)

1. if \(|I| > k\)
2.       for each split dim \(j\) and split value \(s\)
3.             Set   \(I_{j, s}^{+}=\left\{i \in I \mid x_j^{(i)} \geq s\right\}\)
4.             Set   \(I_{j, s}^{-}=\left\{i \in I \mid x_j^{(i)}<s\right\}\)
5.             Set.  \(\hat{y}_{j, s}^{+}=\)average \(_{i \in I_{j, s}^{+}}  y^{(i)}\)
6.             Set   \(\hat{y}_{j, s}^{-}=\)average \(_{i \in I_{j, s}^{-}}  y^{(i)}\)
7.             Set   \(E_{j, s}=\sum_{i \in I_{j, s}^{+}}\left(y^{(i)}-\hat{y}_{j, s}^{+}\right)^2+\sum_{i \in I_{j, s}^{-}}\left(y^{(i)}-\hat{y}_{j, s}^{-}\right)^2\)
8.       Set   \(\left(j^*, s^*\right)=\arg \min _{j, s} E_{j, s}\)
9. else
10.       Set   \(\hat{y}=\) average \(_{i \in I}  y^{(i)}\)
11.       return \(\operatorname{LEAF}\)(leave_value=\(\hat{y})\)
12. return \(\operatorname{Node}\left(j^*, s^*, \operatorname{BuildTree}\left(I_{j^*, s^*}^{-}, k\right), \operatorname{BuildTree}\left(I_{j^*, s^*}^{+}, k\right)\right)\)

\(\operatorname{BuildTree}(I;k)\)

1. if \(|I| > k\)
2.       for each split dim \(j\) and split value \(s\)
3.             Set   \(I_{j, s}^{+}=\left\{i \in I \mid x_j^{(i)} \geq s\right\}\)
4.             Set   \(I_{j, s}^{-}=\left\{i \in I \mid x_j^{(i)}<s\right\}\)
5.             Set.  \(\hat{y}_{j, s}^{+}=\)average \(_{i \in I_{j, s}^{+}}  y^{(i)}\)
6.             Set   \(\hat{y}_{j, s}^{-}=\)average \(_{i \in I_{j, s}^{-}}  y^{(i)}\)
7.             Set   \(E_{j, s}=\sum_{i \in I_{j, s}^{+}}\left(y^{(i)}-\hat{y}_{j, s}^{+}\right)^2+\sum_{i \in I_{j, s}^{-}}\left(y^{(i)}-\hat{y}_{j, s}^{-}\right)^2\)
8.       Set   \(\left(j^*, s^*\right)=\arg \min _{j, s} E_{j, s}\)
9. else
10.       Set   \(\hat{y}=\) average \(_{i \in I}  y^{(i)}\)
11.       return \(\operatorname{LEAF}\)(leave_value=\(\hat{y})\)
12. return \(\operatorname{Node}\left(j^*, s^*, \operatorname{BuildTree}\left(I_{j^*, s^*}^{-}, k\right), \operatorname{BuildTree}\left(I_{j^*, s^*}^{+}, k\right)\right)\)

\(\operatorname{BuildTree}(I;k)\)

1. if \(|I| > k\)
2.       for each split dim \(j\) and split value \(s\)
3.             Set   \(I_{j, s}^{+}=\left\{i \in I \mid x_j^{(i)} \geq s\right\}\)
4.             Set   \(I_{j, s}^{-}=\left\{i \in I \mid x_j^{(i)}<s\right\}\)
5.             Set.  \(\hat{y}_{j, s}^{+}=\)average \(_{i \in I_{j, s}^{+}}  y^{(i)}\)
6.             Set   \(\hat{y}_{j, s}^{-}=\)average \(_{i \in I_{j, s}^{-}}  y^{(i)}\)
7.             Set   \(E_{j, s}=\sum_{i \in I_{j, s}^{+}}\left(y^{(i)}-\hat{y}_{j, s}^{+}\right)^2+\sum_{i \in I_{j, s}^{-}}\left(y^{(i)}-\hat{y}_{j, s}^{-}\right)^2\)
8.       Set   \(\left(j^*, s^*\right)=\arg \min _{j, s} E_{j, s}\)
9. else
10.       Set   \(\hat{y}=\) average \(_{i \in I}  y^{(i)}\)
11.       return \(\operatorname{LEAF}\)(leave_value=\(\hat{y})\)
12. return \(\operatorname{Node}\left(j^*, s^*, \operatorname{BuildTree}\left(I_{j^*, s^*}^{-}, k\right), \operatorname{BuildTree}\left(I_{j^*, s^*}^{+}, k\right)\right)\)

\(\operatorname{BuildTree}(I;k)\)

1. if \(|I| > k\)
2.       for each split dim \(j\) and split value \(s\)
3.             Set   \(I_{j, s}^{+}=\left\{i \in I \mid x_j^{(i)} \geq s\right\}\)
4.             Set   \(I_{j, s}^{-}=\left\{i \in I \mid x_j^{(i)}<s\right\}\)
5.             Set.  \(\hat{y}_{j, s}^{+}=\)average \(_{i \in I_{j, s}^{+}}  y^{(i)}\)
6.             Set   \(\hat{y}_{j, s}^{-}=\)average \(_{i \in I_{j, s}^{-}}  y^{(i)}\)
7.             Set   \(E_{j, s}=\sum_{i \in I_{j, s}^{+}}\left(y^{(i)}-\hat{y}_{j, s}^{+}\right)^2+\sum_{i \in I_{j, s}^{-}}\left(y^{(i)}-\hat{y}_{j, s}^{-}\right)^2\)
8.       Set   \(\left(j^*, s^*\right)=\arg \min _{j, s} E_{j, s}\)
9. else
10.       Set   \(\hat{y}=\) average \(_{i \in I}  y^{(i)}\)
11.       return \(\operatorname{LEAF}\)(leave_value=\(\hat{y})\)
12. return \(\operatorname{Node}\left(j^*, s^*, \operatorname{BuildTree}\left(I_{j^*, s^*}^{-}, k\right), \operatorname{BuildTree}\left(I_{j^*, s^*}^{+}, k\right)\right)\)

\(\operatorname{BuildTree}(I;k)\)

1. if \(|I| > k\)
2.       for each split dim \(j\) and split value \(s\)
3.             Set   \(I_{j, s}^{+}=\left\{i \in I \mid x_j^{(i)} \geq s\right\}\)
4.             Set   \(I_{j, s}^{-}=\left\{i \in I \mid x_j^{(i)}<s\right\}\)
5.             Set.  \(\hat{y}_{j, s}^{+}=\)average \(_{i \in I_{j, s}^{+}}  y^{(i)}\)
6.             Set   \(\hat{y}_{j, s}^{-}=\)average \(_{i \in I_{j, s}^{-}}  y^{(i)}\)
7.             Set   \(E_{j, s}=\sum_{i \in I_{j, s}^{+}}\left(y^{(i)}-\hat{y}_{j, s}^{+}\right)^2+\sum_{i \in I_{j, s}^{-}}\left(y^{(i)}-\hat{y}_{j, s}^{-}\right)^2\)
8.       Set   \(\left(j^*, s^*\right)=\arg \min _{j, s} E_{j, s}\)
9. else
10.       Set   \(\hat{y}=\) average \(_{i \in I}  y^{(i)}\)
11.       return \(\operatorname{LEAF}\)(leave_value=\(\hat{y})\)
12. return \(\operatorname{Node}\left(j^*, s^*, \operatorname{BuildTree}\left(I_{j^*, s^*}^{-}, k\right), \operatorname{BuildTree}\left(I_{j^*, s^*}^{+}, k\right)\right)\)

\(\operatorname{BuildTree}(I;k)\)

1. if \(|I| > k\)
2.       for each split dim \(j\) and split value \(s\)
3.             Set   \(I_{j, s}^{+}=\left\{i \in I \mid x_j^{(i)} \geq s\right\}\)
4.             Set   \(I_{j, s}^{-}=\left\{i \in I \mid x_j^{(i)}<s\right\}\)
5.             Set   \(\hat{y}_{j, s}^{+}=\)average \(_{i \in I_{j, s}^{+}}  y^{(i)}\)
6.             Set   \(\hat{y}_{j, s}^{-}=\)average \(_{i \in I_{j, s}^{-}}  y^{(i)}\)
7.             Set   \(E_{j, s}=\sum_{i \in I_{j, s}^{+}}\left(y^{(i)}-\hat{y}_{j, s}^{+}\right)^2+\sum_{i \in I_{j, s}^{-}}\left(y^{(i)}-\hat{y}_{j, s}^{-}\right)^2\)
8.       Set   \(\left(j^*, s^*\right)=\arg \min _{j, s} E_{j, s}\)
9. else
10.       Set   \(\hat{y}=\) average \(_{i \in I}  y^{(i)}\)
11.       return \(\operatorname{LEAF}\)(leave_value=\(\hat{y})\)
12. return \(\operatorname{Node}\left(j^*, s^*, \operatorname{BuildTree}\left(I_{j^*, s^*}^{-}, k\right), \operatorname{BuildTree}\left(I_{j^*, s^*}^{+}, k\right)\right)\)

\(\operatorname{BuildTree}(I;k)\)

1. if \(|I| > k\)
2.       for each split dim \(j\) and split value \(s\)
3.             Set   \(I_{j, s}^{+}=\left\{i \in I \mid x_j^{(i)} \geq s\right\}\)
4.             Set   \(I_{j, s}^{-}=\left\{i \in I \mid x_j^{(i)}<s\right\}\)
5.             Set   \(\hat{y}_{j, s}^{+}=\)average \(_{i \in I_{j, s}^{+}}  y^{(i)}\)
6.             Set   \(\hat{y}_{j, s}^{-}=\)average \(_{i \in I_{j, s}^{-}}  y^{(i)}\)
7.             Set   \(E_{j, s}=\sum_{i \in I_{j, s}^{+}}\left(y^{(i)}-\hat{y}_{j, s}^{+}\right)^2+\sum_{i \in I_{j, s}^{-}}\left(y^{(i)}-\hat{y}_{j, s}^{-}\right)^2\)
8.       Set   \(\left(j^*, s^*\right)=\arg \min _{j, s} E_{j, s}\)
9. else
10.       Set   \(\hat{y}=\) average \(_{i \in I}  y^{(i)}\)
11.       return \(\operatorname{LEAF}\)(leave_value=\(\hat{y})\)
12. return \(\operatorname{Node}\left(j^*, s^*, \operatorname{BuildTree}\left(I_{j^*, s^*}^{-}, k\right), \operatorname{BuildTree}\left(I_{j^*, s^*}^{+}, k\right)\right)\)

\(\operatorname{BuildTree}(I;k)\)

1. if \(|I| > k\)
2.       for each split dim \(j\) and split value \(s\)
3.             Set   \(I_{j, s}^{+}=\left\{i \in I \mid x_j^{(i)} \geq s\right\}\)
4.             Set   \(I_{j, s}^{-}=\left\{i \in I \mid x_j^{(i)}<s\right\}\)
5.             Set   \(\hat{y}_{j, s}^{+}=\) majority \(_{i \in I_{j, s}^{+}}  y^{(i)}\)
6.             Set   \(\hat{y}_{j, s}^{-}=\) majority \(_{i \in I_{j, s}^{-}}  y^{(i)}\)
7.             Set   \(E_{j, s} = \frac{\left|I_{j, s}^{-}\right|}{|I|} \cdot H\left(I_{j, s}^{-}\right)+\frac{\left|I_{j, s}^{+}\right|}{|I|} \cdot H\left(I_{j, s}^{+}\right)\)
8.       Set   \(\left(j^*, s^*\right)=\arg \min _{j, s} E_{j, s}\)
9. else
10.       Set   \(\hat{y}=\) majority \(_{i \in I}  y^{(i)}\)
11.       return \(\operatorname{LEAF}\)(leave_value=\(\hat{y})\)
12. return \(\operatorname{Node}\left(j^*, s^*, \operatorname{BuildTree}\left(I_{j^*, s^*}^{-}; k\right), \operatorname{BuildTree}\left(I_{j^*, s^*}^{+}; k\right)\right)\)

The only change from regression to classification:

• Line 5, 6, 10, average becomes majority vote
• Line 7 error more involved 

• \({I}\)  = 9, \(\left|I_{j, s}^{-}\right|\) = 6, \(\left|I_{j, s}^{+}\right|\) = 3
• So, \(E_{j, s} = \frac{6}{9} H\left(I_{j, s}^{-}\right) +\frac{3}{9}  H\left(I_{j, s}^{-}\right)\)

\(H\left(I_{j, s}^{-}\right) = -[\frac{3}{6} \log _2\left(\frac{3}{6}\right)+\frac{2}{6} \log _2\left(\frac{2}{6}\right)+\frac{1}{6} \log _2\left(\frac{1}{6}\right)]\)

\(H\left(I_{j, s}^{+}\right) = -[\frac{1}{3}  \log \left(\frac{1}{3}\right)+\frac{0}{3} \log _2\left(\frac{0}{3}\right)+\frac{2}{3} \log _2\left(\frac{2}{3}\right)]\)

\(H=-\sum_{\text {class }_c} \hat{P}_c (\log _2 \hat{P}_c)\)

Bagging

Bagging

Nearest neighbor classifier

• for a new data point \(x_{new}\) do:
  • find the \(k\) points in training data nearest to \(x_{new}\) 
    • For classification: predict label \(\hat{y_{new}}\) for \(x_{new}\) by taking a majority vote of the \(k\) neighbors's labels \(y\)
    • For regression: predict label \(\hat{y_{new}}\) for \(x_{new}\) by taking an average over the \(k\) neighbors' labels \(y\)

• Hyperparameter: \(k\)
• Also need
  • Distance metric (typically Euclidean or Manhattan distance)
  • A tie-breaking scheme (typically at random)