From random forests to regulation: interpreting supervised learners to guide biological discovery

Karl Kumbier

September 12, 2022

 

 

In search of mechanistic insights into development and function

  • Genomes contain vast amounts of information (e.g. 6M variants in 20K human genes)

 

  • Complex biological processes are the result of interacting components 

 

  • Linking genomic processes to biological outcomes requires the exploration of enormous spaces

Domain insights through supervised learning

Natural phenomena

ML models

Domain insights

?

Prediction

Interpretation

"Accuracy generally requires more complex prediction methods."

​     - Leo Breiman, Statistical Modeling: The Two Cultures (2001)

Outline

  • From genomic to statistical interactions​

  • Market baskets and genomics​

  • Iterative random forests​ & signed iterative random forests

  • Case studies: Interaction discovery in Drosophila and UK Biobank cohort

From genomic to statistical interactions

Embryonic development in Drosophila

0-1:20 hours

1:20-3:00 hours

3:00-3:40 hours

3:40-5:20 hours

5:20-9:00 hours

9:20-16:00 hours

image: Volker Hartenstein

images: BDGP

 Kr expression

Enhancers regulate spatio-temporal programs of gene expression

Enhancers: segments of the genome that coordinate transcription factor (TF) activity to regulate gene expression.

Experimental evaluation of enhancer elements in Drosophila

Pfeiffer et al. (2008)

even-skipped

expression

wt

transgenic

Experimental evaluation of enhancer elements in Drosophila

Hiromi et al. (1985), Harding et al. (1989), Goto et al. (1989), Pfeiffer et al. (2008)

even-skipped expression

wt

transgenic

Identifying regulatory interactions from high-throughput genomic data

  • Fisher (epistasis): "deviation from the addition of superimposed effects (...) between different Mendelian factors."
  • Traditionally formulated as a multiplicative interaction term (e.g. in logistic regression)

 

 

From genomic to statistical interactions: classical interpretation

logit(p) = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_{12} x_1 x_2

Problems

  • Non additivity depends on response scaling
  • Computationally intractable for high-order interactions
  • TF interactions are not necessarily multiplicative

 

High-order interactions at enhancer elements drive embryonic development

Goto et al. (1989), Harding et al. (1989), Small et al. (1992), Isley et al. (2013), Levine et al. (2013)

From genomic to statistical interactions: our interpretation

r(\mathbf{x})=
\prod_{j\in\text{A}} 1(x_j > t_j)
\cdot
\prod_{j\in\text{R}} 1(x_j \le t_j)

activators

repressors

\mathbf{x}=(x_1, \dots, x_p)

Segment of the genome

DNA binding for p transcription factors (TFs)

Order-s interaction: s = #activators + #repressors

r(\mathbf{x}) = 1
x_{bcd}
x_{cad}
\dots

RuleFit: rule-based interaction discovery (Friedman and Popescu, 2008)

  1. Identify a collection of marginally important features
  2. Search for predictive order-2 rules among marginally important features
\dots

Computational costs grow as

O(p^s)

Misses interactions with weak  marginal effects

image: Lee and Haber (2014)

Thresholding rules define expression domains

Chopra and Levine (2009)

Dl +

Dl -

Wolpert (1968), Jaeger and Reinitz (2006), Chopra and Levine (2009), Zizen et al. (2009), Knowles and Biggin (2013), Levine (2013), Staller al. (2015), ...

Jaeger and Reinitz (2009)

From genomic to statistical interactions

\mathbb{P}_n(y = 1 | S)
\mathbb{P}_n(S|y = 1)

(1) How precisely does an interaction predict class-1 observations?

(2) How prevalent is an interaction among class-1 observations?

S = (x_{j_1} > t_{j_1}) \,\&\, \dots \,\&\, (x_{j_s} \le t_{j_s})

Interactions:

Responses:

y \in \{0, 1\}
r(\mathbf{x})=

?

Market baskets and genomics 

Interactions in market baskets

What combinations of items do customers purchase together?

Interactions in market baskets

What combinations of items do customers purchase together?

What combinations of items do different types of customers purchase together?

Interactions in market baskets

Z_1
Z_2
Z_3
Z_4
\mathcal{I}_1
\mathcal{I}_2
\mathcal{I}_3
\mathcal{I}_4

Feature-index sets

Random intersection trees (RIT)

Shah and Meinshausen (2014)

Leverage sparsity in market baskets to search for frequently co-occurring items in a computationally efficient manner

  1. Randomly sample feature index sets from class-C observations:
  2.  Intersect sampled feature index sets in a tree like fashion up to depth D
  3. Return all feature combinations that "survive" intersection procedure up to depth D
\{\mathcal{I}_i \subseteq\{1, \dots, p\}: Z_i = C\}

Randomly sampled

class-C observation

"survived" interaction

Random intersection trees (RIT)

Shah and Meinshausen (2014)

Random intersection trees (RIT)

Shah and Meinshausen (2014)

Random intersection trees (RIT)

Shah and Meinshausen (2014)

Genomic response

Genomic features

Translating the market basket problem into genomics

\dots

Genomic response

Genomic features

Challenges:

  1. Genomic features are typically measured in concentrations/counts
  2. Binding does not imply regulation (Li et al. 2008)
\dots

Translating the market basket problem into genomics

iterative Random Forests (iRF)

&

signed iterative Random Forests (siRF)

Joint work with Sumanta Basu, James B. Brown, and Bin Yu

Iterative Random Forest to identify high-order interactions in genomic data

  1. Iteratively re-weighted Random Forests stabilize decision path                       
  2. Generalized random intersection trees search for high-order interactions         
  3. Stability bagging evaluates interactions                           

Iterative Random Forests (iRF) build on predictability, computability, and stability to identify genomic interactions in developing Drosophila embryos

Open source R implementation: github.com/karlkumbier/iRF2.0

Iteratively re-weighted random forests

Classification and regression trees (CART)

Breiman et al. (1984)

 

For current node:

  1. Select splitting feature and threshold
  2. Partition data 
  3. Repeat until stopping criteria
x_2>t_2
x_3>t_3
x_4>t_4

Classification and regression trees (CART)

Breiman et al. (1984)

 

For current node:

  1. Select splitting feature and threshold
  2. Partition data 
  3. Repeat until stopping criteria
x_2>t_2
x_3>t_3
x_4>t_4
(x_2 > t_2) \,\&\,(x_3 > t_3) \,\&\,(x_4 > t_4)

Random Forests

Breiman (2001)

Random forests modify CART to improve predictive accuracy:

  1. CART trees are trained on bootstrap samples of the data
  2. CART criterion evaluated on subset of features sampled uniformly at random

Random forest modifications improve generalization but reduce stability!

Feature-weighted Random Forests

Amaratunga et al. (2008)

Random forests: 

At each node of the decision tree, uniformly sample a subset of features

Feature-weighted random forests: 

At each node of the decision tree, sample a subset of features with probability proportional to 

w\in\mathbb{R}^p_+
w_1
w_2
w_3
w_4
w_5
w_1
w_2
w_3
w_4
w_5

Feature weights

The CART criterion: Gini impurity

(\pi, N)
(\pi_l, N_l)
(\pi_r, N_r)

Proportion positive responses

Number of observations

Gini impurity:

I_G(\pi) = \pi (1-\pi)

Decrease in Gini impurity:

I_G(\pi)-\frac{N_l}{N}I_G(\pi_l) - \frac{N_r}{N}I_G(\pi_r)

Mean decrease in impurity:

On average, how much does splitting on a variable decrease the Gini impurity?

Iterative re-weighting stabilizes random forest decision paths

Gini importance

Iteration 1

Iteration K

w_1
w_2
w_3
w_4
w_5
w_1
w_2
w_3
w_4
w_5

Feature weights

Iterative re-weighting helps recover high-order interactions

Generalized random intersection trees

Encoding decision paths to extract active features

Active

Inactive

Continuous measurements

Binary features

Encoding decision paths to extract enriched and depleted features

Enriched

Depleted

Generalized random intersection trees search for high-order interactions

\cap
\cap
\cap
\cap
\{1,4\}

1. Iteratively re-weighted random forests

3. RIT on random forest decision paths

2. Decision path feature transformation

\mathbf{x}_1 \rightarrow \mathcal{I}_1 =
\mathbf{x}_n \rightarrow \mathcal{I}_n =

.

.

.

\emptyset

Encoding decision paths to extract prevalent decision rules

Continuous measurements

Binary feature encoding

Decision rules

Encoding decision paths to extract prevalent decision rules

\mathbf{x}_i
S_{i_1}
S_{i_T}

. . .

Generalized random intersection trees search for high-order interactions

Prevalent interactions

Binary feature encoding

RIT

. . .

Runtime comparison between iRF and RuleFit

Evaluating interactions

PCS framework for veridical data science

(Yu and Kumbier, 2020)

Predictability  (from ML, Stats) evaluates whether models/results reflect external reality

 

Computability (from ML) enables domain-inspired simulations to compare against known structure

 

Stability  (from Stats) assesses the reproducibility of results relative to data and model perturbations

 

The PCS framework unifies and expands on ideas from statistics and machine learning

Precision measures the predictive accuracy of an interaction across an RF

Precision:

P(y=1|S)=\frac{1}{T} \sum_{t=1}^T \frac{\sum_{i=1}^n I(S\subseteq \mathcal{I}_{i_t}) \cdot I(y_i = 1)}{\sum_{i=1}^n I(S\subseteq \mathcal{I}_{i_t})}
P(y=1|\{1, 4\}) = 4/6
P(y=1|\{1, 3, 4\}) = 3/4

Examples:

Prevalence measures the stability of an interaction across an RF

Prevalence: 

P(S|y=1)=\frac{1}{T} \sum_{t=1}^T \frac{\sum_{i=1}^n I(S\subseteq \mathcal{I}_{i_t}) \cdot I(y_i = 1)}{\sum_{i=1}^n I(y_i = 1)}
P(\{1, 3, 4\}|y = 1) =3/5
P(\{1, 4\}|y = 1) =4/5

Examples:

Null metrics describe importance measures under simple structure computed from RF

Biology: TF enrichment among active enhancers

Null model: prevalence among inactive elements

Biology: cooperative binding among TFs

Null model: expected prevalence under independent selection

Biology: functional binding v. inactive binding

Null model: precision of interaction subsets

Bagging evaluates stability of interactions across entire iRF workflow relative to resampling

\cap
\cap
\cap
\cap
\cap
\cap
\cap
\cap
\{1,4\}
\{1,3\}
P(S|y=1)
P(S|y=1)
\dots

1. Iteratively re-weighted RF stabilize decision paths

2. gRIT searches for high-order interactions along decision paths

3. Importance metrics evaluate interactions in fitted RF

Outer layer bootstrap samples

\emptyset
\{1,4\}

Discovering "functional" TF binding and interactions in the Drosophila embryo

Joint work with Sumanta Basu, James B. Brown, Susan Celniker, Erwin Frise, and Bin Yu

Predicting enhancer activity throughout embryonic development

Enhancers: Pfeiffer et al. 2008, Fisher et al. 2012, Kvon et al. 2014

ChIP: MacArthur et al. 2009, Li et al. 2008, modENCODE/modERN consortia

x_1
x_2
\dots
y

iRF predicts stage-specific enhancer activity with high accuracy and recovers well-known pairwise interactions

Early stage (not shown): 24 TFs; Basu, K., Brown, and Yu (2018)

 

All stages (shown): 307 TFs; K., Basu, Brown, Celniker, Frise and Yu

Predicting enhancer activity throughout embryonic development

Early stage (not shown): 24 TFs; Basu, K., Brown, and Yu (2018)

 

All stages (shown): 307 TFs; K., Basu, Brown, Celniker, Frise and Yu

iterative Random Forests recover well-known pairwise interactions

Novel order-3 interactions exhibit AND-like behavior (Hth, Kni, Kr)

Novel interactions show strong concordance with temporal expression

siRF correctly identifies known regulators of eve

Binding of known regulators correctly identified by siRF and missed by IDR

Binding with no reported function identified by IDR and not called by siRF

siRF filters down to high-quality set of  functional peaks

siRF functional binding GO term enrichment

IDR binding GO term enrichment

Figure: Wotton et al. (2015)

Gap gene network as validation

EpiTree pipeline for detecting epistatic interactions in the UK Biobank

Joint work with: Merle Behr, Aldo Cordova-Palomera, Matthew Aguirre, Euan Ashley, Atul J. Butte, Rima Arnaout, Ben Brown, James Priest, Bin Yu

New method EpiTree for epistasis discovery

  • Flexible and non-linear mathematical form

 

  • Suited to detect interactions of order > 2
  • Entirely genetic
  • Governed by epistasis (Morgan et al. 2018)
  • Common trait

Positive control phenotype: red hair

 

  • ~500,000 individuals
  • Self-reported hair color
  • 10,000,000 variants from array genotype data

UK Biobank

PCS inference for epistatic interactions

Learn models

Inference

Predicting red hair in the UK biobank

  • Recovers known genetic determinants of hair color & pigmentation
  • Interactions recapitulate results from Morgan et al. (2018)
  • EpiTree does not require a priori knowledge of important/causal variants

siRF interactions capture non-linearities missed by multiplicative interactions

Next steps: capturing heterogeneity in ALS through localization

  • Amyotrophic lateral sclerosis (ALS) - fatal, neurodegenerative.

  • Over 25 known genetic causes. 90% of cases are sporadic (SALS); many of the familial (FALS) cases also have unknown cause

  • No effective treatments exist

 

Can we accelerate drug discovery by learning ALS subtypes and the patterns of dysregulation that define them?

Joint work with: Julia Lazzari-Dean, Maike Roth, Steven Altschuler, and Lani Wu

Figure: Zou, Z. Y. et al. (2017)

 

Summary

  • iRF and siRF identify well known interactions in Drosophila and UK biobank data and posit new, high-order interactions.
  • By decoupling interaction order from the computational cost of discovery, iRF and siRF allow us to investigate mechanisms in genome biology and beyond. 

Ackowledgements

Drosophila

Sumata Basu, Erwin Frise, Susan Celniker, James B. Brown, Bin Yu

UK biobank

Merle Behr, Aldo Cordova-Palomera, Matthew Aguirre, Euan Ashley, Atul J. Butte, Rima Arnaout, Ben Brown, James Priest, Bin Yu

Thank You!

Made with Slides.com