Book 3. Operational Risk

FRM Part 2

OR 16. Case Study: Model Risk and Model Validation.pdf

Presented by: Sudhanshu

Module 1. Model Risk and Model Validation

Module 1. Model Risk and Model Validation

Topic 1. Model Risk Exposure

Topic 2. Model Risk Management

Topic 3. Model Risk Case Studies

Topic 1. Model Risk Exposure

  • Definition: A quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates.

    • Generates an estimate or forecast using inputs, an underlying theory, and a set of assumptions.

    • Inputs can be quantitative or qualitative.

    • Model outputs are estimates, not definitive, and are subject to estimation error.

  • Types of Model Risk
    • ​Execution Risk: Models not functioning as intended due to errors in input data or coding.
    • Conceptual Errors: Assumptions are invalid (do not represent reality) and incorrect modeling techniques are used.
  • ​More difficult to identify than execution risk.

  • A model that works in some economic environments may not perform well in others.

  • Model assumptions and limitations should be thoroughly documented and communicated.

  • Example: Risk measurement models relying on correlations may underestimate correlations during volatile market conditions.

Practice Questions: Q1

Q1. Which one of the following items is least likely associated with a model?

A. Qualitative inputs.

B. Mathematical theories.

C. Precise output.

D. Assumptions.

Practice Questions: Q1 Answer

Explanation: C is correct.

Model outputs are forecasts or estimates, which are not precise. Model inputs can be qualitative or quantitative. Also, models rely on a set of assumptions, and use economic and mathematical theories.

Topic 2. Model Risk Management

  • Role of the MRM Team: Comprises experts independent of original model developers. Charged with mitigating model risk and sets standards for

    • Model version control

    • Data quality

    • Model documentation

  • Model Risk Tiers: Not all models pose the same organizational risks; scrutiny should be proportional to risk level. Level of model risk tier depends on:
    • Materiality of model output: (e.g., dollar value of loss if the model fails).
    • Model complexity

    • Client-facing aspect: Whether the model will be client-facing.

    • Regulatory compliance: Whether the model is used for regulatory compliance.

  • Highest tiers require more frequent validation (every 2-3 years) and comprehensive backtesting.

  • All models should be reviewed annually for changes in environment, input data quality, etc.

  • ​Best Practices in MRM and Validation
    • ​MRM should be a continuous, ongoing process, not just a periodic review.

    • MRM teams review periodic model performance reports from model owners.

    • The existence of an MRM team should not lead to complacency among model users and developers, who are the first line of defense.

Practice Questions: Q2

Q2. Which of the following statements describes model execution risk?

A. Inaccurate model inputs.

B. Model coding that is consistent with model assumptions.

C. Inappropriate model assumptions.

D. Incorrect modeling techniques.

Practice Questions: Q2 Answer

Explanation: A is correct.

Execution risk arises due to errors in input data or in coding of the model.

Topic 3. Model Risk Case Studies

  • Case Study 1: Gaussian Copula and CDO Pricing

    • Model: David X. Li's Gaussian copula function for pricing Collateralized Debt Obligations (CDOs).

    • Assumption: Markets were efficient, and Credit Default Swap (CDS) prices correctly set by the market. Assumed static (constant) asset correlations.

    • Problem: Historical correlations of residential mortgage defaults were low, but as housing prices fell in 2008, CDS prices (and implied correlations) shot up. The model incorporated changing correlations with a lag.

    • Outcome: Collapse of the CDO market. The model encouraged ignoring real-world randomness and uncertainty.

    • Lesson Learned (MRM Context): Increase transparency regarding assumptions and limitations. Effective communication is critical, especially when users lack quantitative background.

  • Case Study 2: Barclays Acquisition of Lehman Brothers' Assets and Spreadsheet Error

    • Scenario: Barclays bid for Lehman Brothers' assets during liquidation.

    • Error: A junior law associate converted a spreadsheet with hidden rows (representing assets Barclays did not want) into a PDF, revealing the hidden rows.

    • Problem Type: Implementation error.

    • Outcome: Barclays was legally bound to contracts they didn't intend to bid on, leading to potential large losses.

    • Lesson Learned: Highlights the risk of implementation errors, even with non-model software. Rigorous testing of inputs and processes is crucial.

  • Case Study 3: NASA Mars Orbiter

    • Scenario: Loss of a $125 million satellite.

    • Error: The engineering team at Lockheed Martin used English units of measurement, while NASA's convention was the metric system.

    • Problem Type: Input measurement error.

    • Outcome: Loss of the satellite due to inaccurate model inputs.

    • Lesson Learned: Rigorous testing of all assumptions and inputs is paramount. Small mistakes can lead to very large losses, emphasizing the need for robust MRM.

Topic 3. Model Risk Case Studies

Practice Questions: Q3

Q3. What is the most likely reason for the failure of the Gaussian copula function to price CDOs?

A. Model computation error.

B. Inappropriate model for the task.

C. Invalid model assumption.

D. Invalid use of model output.

Practice Questions: Q3 Answer

Explanation: C is correct.

The Gaussian copula function relied on the Gaussian constant input, which was based on an assumption of static (constant) asset correlations in a collateral pool.

Practice Questions: Q4

Q4.  The Barclays bankruptcy court bid for Lehman assets most likely suffered from:

A. ongoing monitoring.

B. improper model use.

C. invalid model assumptions.

D. implementation error.

Practice Questions: Q4 Answer

Explanation: D is correct.

The failure to delete spreadsheet rows representing assets that Barclays did not want to bid on represented an implementation error.

Practice Questions: Q5

Q5. The failure of NASA's Mars Orbiter mission can be directly attributed to:

A. model inaccuracy.

B. model assumptions.

C. incorrect model choice.

D. incorrect model inputs.

Practice Questions: Q5 Answer

Explanation: D is correct.

The units of measurement for model inputs were in accurate, which led to the failure of the model and a very large loss.