Notes from the Trenches,

How Good Models Go Bad

Statistical & Machine Learning Approaches to Investment Management May 2019

©2019 Economic Data Sciences

# Agenda

©2019 Economic Data Sciences

1. Common pitfalls & how to avoid them
2. Instability & Interpretation
• A Principal Components Example
• A Kalman Filter Example
3. More than one way to scramble an egg
• Features and interpretation over model selection
4. A Focus on Implementation & Implied Assumptions

# Common Pitfalls

©2019 Economic Data Sciences

### The devil is in the details

*Courtesy of Caitlin Hudon https://twitter.com/beeonaposy

# Avoiding Common Pitfalls

©2019 Economic Data Sciences

### No magic formula, but there are some good guidelines

• Reproducing sub-results in different environments (Excel & R/MatLab)
• Interpret, decompose, interpret
• No replacement for time, experience, & care
• You will be rushed -- It is okay to delay
• Review your assumptions, then do it again
• Are there implicit assumptions?
• Can you realistically implement?
• Was all data available? revised? realistic lag time?

# Instability & Interpretation

©2019 Economic Data Sciences

### Stability & Interpretation normally go hand in hand

• As humans, we are natural story-tellers
• It is far too easy to append a narrative to data or output
• But if that output is unstable, how can our narrative be true?
• Ensuring our analysis is stable and consistent is one of the most important excercises we can carry out
• Let's look at two examples:
• Principal Component Analysis
• Kalman Filter & Neural Nets

# PCA Intro 1

©2019 Economic Data Sciences

### Principal Component Analysis -- A Quick Primer

• In contrast to Linear Regression which finds the best β to fit                 ,    PCA find the best combination of r so that                 and
• Like Linear Regression where betas are 1, moves the data instead
• What are the principal components?
1. The first principal component is the combination of data that explain the most variance and
2. Every following component is the same, but also with the covariance between it and the previous component equal to zero
\bold y = \pmb\beta*\bold r
\bold y = \pmb\beta*\bold r
\pmb\beta'*\pmb\beta = 1
\pmb\beta'*\pmb\beta = 1

# PCA Intro 2

©2019 Economic Data Sciences

### Principal Component Analysis -- What Is It Good For?

• Dimension Reduction & Clustering

• In practice
• Check autocorrelation of returns
• Rotation usually is helpful
• Can be used/compared with Factor Models to gain intuition

Explain All This

With Just This

# PCA Problems

©2019 Economic Data Sciences

# Kalman Filter Intro 1

©2019 Economic Data Sciences

### Kalman Filter -- A Quick Primer

• The Kalman Filter is a model with many names
• First consider Time-Series Model, the Vector Auto-Regression

• The prediction for      is a state, which helps prediction

• The state is sometimes considered a hidden regime or layer
x_{t} = \phi_{t-1} + w_{t}
x
y_{t} = \Alpha_{t}x_t + v_{t}

# Kalman Filter Intro 2

©2019 Economic Data Sciences

### The importance of the transition matrix

• When using the Kalman Filter for time-varying models and states, the transition matrix helps stability of the state
• First consider two states

\begin{matrix} S_{1,t} \\ S_{2,t} \end{matrix} = \pmb\phi_{t-1} + \bold w_{t}
 95% 5% 10% 90%
\begin{matrix} S_{1} \\ S_{2} \end{matrix}
S_{1}
S_{2}

From

State

To

State

# Kalman Filter Regimes

©2019 Economic Data Sciences

Unstable

# More Complexity Adds Difficulty

©2019 Economic Data Sciences

# Using Theory With Regimes

©2019 Economic Data Sciences

# Comparison With Neural Nets

©2019 Economic Data Sciences

### A simple visual comparison

Hidden Markov Model

Inputs

Hidden States Selected on Probability of Fit According to Features

States Effect Output

Neural Net

Select Features That Describe Hidden Layer

Hidden Layers Chosen

Hidden Layers Effect Output

# Comparison Of Output

©2019 Economic Data Sciences

# Implementation

©2019 Economic Data Sciences

### Let's take a step back and discuss implementation

• One of the most famous models in finance is the Shiller PE 10

*Data from EDS & Robert Shiller's website

# A Simple Implementation

©2019 Economic Data Sciences

### Implementation of this signal, on its face, should be easy

*Data from EDS & Robert Shiller's website

# Quick Test

©2019 Economic Data Sciences

### However, implementing a quick test, we don't see gains?

• This test implies free trading costs

*Data from EDS, Shiller & Bloomberg, courtesy of London Business School

# So What Went Wrong Part 1

©2019 Economic Data Sciences

### Implied Assumptions -- If You Sell Equities, What Do You Buy?

• Bonds share the same relationship with the measure

Interesting Side Note, These Are Driven By Inlfationary Periods

*Data from EDS, Shiller & Bloomberg, courtesy of London Business School

# So What Went Wrong Part 2

©2019 Economic Data Sciences

### Didn't Check Assumptions

• Strong autocorrelation, breaks an underlying assumption of OLS
• Interpretation: A new portfolio created each month?

*Data from EDS & Shiller

# Different Models Same Insight

©2019 Economic Data Sciences

### Both MVO & OLS Lead to the Same Output

• Let's go through the spreadsheet here

Unfortunately, due to data restrictions I am unable to make this spreadsheet public. Please contact me for more details.

*Data from EDS & Bloomberg, courtesy of London Business School

# From Slack      L1      L2

©2019 Economic Data Sciences

### Generalized Additive Models - Mixing Elastic Net with Cross-Validation

• Similar to the Slack variable we have a clear intuition
• L1-Norm: Make sure outliers don't drive model
• L2-Norm: Which factors should we choose?
\small \boldsymbol L(\lambda_1,\lambda_2,\beta) = \underbrace{\text{\textbar} y - X\beta \text{\textbar}^2}_{OLS} + \underbrace{\lambda_2\text{\textbar} \beta \text{\textbar}^2}_{L2-Norm} + \underbrace{\lambda_1 \text{\textbar} \beta \text{\textbar}_1}_{L1-Norm}

# Using Cross-Validation

©2019 Economic Data Sciences

### Cross-Validation is used to choose parameters

• The intuition behind this approach is sacrificing the 'best fit' for stability
• A New Goal: The least wrong, the most often

1st Iteration

2nd Iteration

10th Iteration

...

\overbrace{\textcolor{#f9f9f9}{..............................}}^{\text Training Folds}
\overbrace{\textcolor{#f9f9f9}{}}^{\text Test Fold}
\rArr E_1
\rArr E_2
\rArr E_{10}
\Huge \rbrace \small Avg

Full Training Data

# Why Are GAMs Useful?

©2019 Economic Data Sciences

### Rather than focusing on a new model/approach that is harder to interpret, GAMs provide a framework for robust traditional modelling

• At the end of the day, GAMs are just a special case of OLS
• But then, so are most models
• GAMs are a machine learning framework to automate the model building process
• 'Features are King' -- but which ones to choose?
• These features should be interpretable, robust, and consistent
• Easier to understand the connection between theory and output

# Main Takeaways

©2019 Economic Data Sciences

### Focus on details & assumptions, interpretation, implementation, and consistency

• Avoid pitfalls through reproducibility, patience, and care
• Learn your tools and systems, check the details and data types
• By focusing on interpretation all other good habits follow
• What does this mean in terms of implementation?
• What are the underlying & implicit assumptions?
• Do I understand and can I explain the output?
• Is it consistent? Am I achieving my goals?

# Frameworks Not Models

©2019 Economic Data Sciences

### Thinking of these in terms of frameworks will gain more benefits

• Building a model is great, but it isn't really a solution
• Successful value add will come by combining many tools and insights into a robust framework for decision making
• These tools are about automation & enablement
• Focus more on finding implementable & interpretable insights
• Look more broadly, more deeply than before
• This is fundamental

# Why Bother With All This? Part 1

©2019 Economic Data Sciences

### More informed decisions, in a more robust process, leads to better results

• Better outcomes, one example of many is in Hedge Funds

*Data from EDS & Bloomberg, courtesy of London Business School

# Why Bother With All This? Part 2

©2019 Economic Data Sciences

### Using A.I. in financial services is becoming a 'must-have'

• Aug 2017: 20% usage of A.I. or Machine Learning
• Aug 2018: 56% usage

### Exploratory programs on machine learning

• A.I. based risk management
• Dynamic factor analysis using A.I.
• A.I. reconciling investment decisions

# Concluding Remarks

©2019 Economic Data Sciences

### More informed decisions, in a more robust process, leads to better results

• ML and AI aren't neccessarily a value-add
• When combined in a strong framework the value adds are fundamental:
• Enabling you to work with a more holistic perspective
• Increasing efficiency and therefore productivity
• Allows you to focus more on insights & implementation
• Know your systems, big problems require an understanding of tools
• Interpretation is the realm of the human
• These tools are powerful, but if you can't understand the 'why' then they are little use to anyone

# Disclaimers

©2019 Economic Data Sciences

Please remember that past performance may not be indicative of future results. Different types of investments involve varying degrees of risk, and there can be no assurance that the future performance of any specific investment, investment strategy, or product made reference to directly or indirectly in this presentation, will be profitable, equal any corresponding indicated historical performance level(s), or be suitable for an individual's portfolio.

Our projections are based on current market conditions which can vary over the coming months and weeks. Additionally, our projections are based on historical market behavior which may vary unexpectedly. Using Machine Learning, our tool should adjust to new market fluctuations but we might not be able to avoid short term volatility.