Home
Week 8

Week 8

Analytical Modeling Overview

Instructors
Maclean Gaulin

What to do with our data?

  • Once we have data, what then?
  • Use it!
  • Model the world
  • Descriptive: What happened?
  • Diagnostic: Why did it happen?
  • Predictive: What will happen?
  • Prescriptive: What should we make happen?

What is a model?

  • A model is a description of how the world works
  • E.g. market equity is discounted cash flows
  • Could be flow-charts to formalize decision-making
  • Example: Lease accounting determination
  • Could be mathematical models
  • Example: Y = m*X + b

Modeling has Pros and Cons

  • Data analytics has profound benefits in Accounting
  • Business insight, cost and revenue forecasting, efficient audit and internal controls, fraud detection, …
  • But models are imperfect
  • Built on assumptions
  • Limited by data
  • Cautious optimism and professional skepticism

General Model Intents

  • Descriptive: What happened?
  • Diagnostic: Why did it happen?
  • Predictive: What will happen?
  • Prescriptive: What should we make happen?
  • Significant overlap (not a clean classification)
  • Because it’s all just modeling

Model Intent Timeline

  • Identify a situation and what occurred (descriptive)
  • Determine why it occurred the way it did (diagnostic)
  • Leverage that knowledge to predict outcomes outside the specific situation (predictive)
  • Consider different outcomes and choose one for some (good) reason (prescriptive)

Descriptive Models

  • Intent is to describe, or capture something
  • Lots of learning about the data and setting
  • Your accounting domain knowledge is critical here!
  • Typically where we do exploratory data analysis
  • Often utilizing summary statistics, simple regressions
  • Example: Assets & Liabilities have R2 of 99%
  • Note! Running a regression at this stage doesn’t imply causality. It just shows us average relationships.

Diagnostic Models

  • Intent is to take the next step and learn why what you observed happened
  • The validity of the results relies entirely on the model
  • This is where most economists live
  • They are notoriously averse to predicting the future!
  • Often done with regressions/classifications
  • Outlier analysis for audit / forensic accounting
  • Causal models to identify true relationships

Predictive Models

  • Intent is to use your knowledge to model new outcomes (i.e. predicting the future)
  • Also used to understand what would have happened if different decisions had been made
  • Very dependent on model and assumptions
  • Was data used “normal” or representative?
  • Will assumptions change going forward (e.g. interest)?
  • Often conflated with diagnostic models

Prescriptive Models

  • Intent is to decide what actions to take
  • Flow-charts to guide decisions
  • Models with turnable dials to perform what-if analyses
  • Can result in a firm conclusion, or a data-backed input for decision-makers
  • Should factor in harder-to-model inputs, like risk
  • Careful about relying on predictive models of causality (the point of choosing is to change things!)

Modeling Caveats

  • Most modeling boils down to using historical data to make inferences about the future
  • Correlational approaches (most machine learning) often cannot predict what it has not “seen”
  • Modeling the structure (most economics) often will not pick up real world complexity, and parameters are determined by correlational approaches
  • It all depends on the assumptions (data and model)

From theory to practice

  • An analytical model is usually defined as a relationship with unknown quantities
  • E.g. Y = m*x + b
  • Fitting is estimating the unknown quantities
  • E.g. the m and b
  • Example: Altman’s Z Score – bankruptcy prediction
  • Z = 1.2 WCap + 1.4 RE + 3.3 EBIT + 1 Rev + 0.6 MVE
  • Z < 1.1 “Bankrupt zone”, Z < 2.6 “Grey Zone”

What is a good fit?

What is a good fit?
  • How well does the model describe the data you have
  • Called in sample fit, based on training data
  • How well does the model predict new data
  • Called out of sample fit, based on test data
What is a good fit?

What is a good fit?

  • With regressions, we often measure how “wrong” the predictions are
  • R2: how much of the outcome variable’s value the model explains
  • RMSE/MAE: measures of average error
What is a good fit?

Fit Examples

Fit Examples

What can data tell us?

  • Anything we measure is definitionally historical
  • Accounting information especially so!
  • The past and the future are often similar, so historical data is often very useful for future decisions
  • But not always
  • In fact, changing the future is often a core pursuit

Overfitting: too much of a good thing

  • Some models can just memorize the training data
  • Does not mean they have learnt the “truth”
Overfitting: too much of a good thing

Identifying and solving Overfitting

  • Split data into training data and testing data
  • Training: fit best model, with potential overfitting
  • Test: model hasn’t seen test data, so higher test error than training suggests overfitting
  • Solutions:
  • More data (synthetic data?)
  • Change training
  • Adjust model (with cross validation)

Identifying and solving Overfitting

Identifying and solving Overfitting

Garbage in, Garbage out

  • Random errors (called noise) results in worse measures and outcomes, but randomly
  • E.g. predicting default risk inaccurately
  • Errors all in the same direction (called bias) can result in consistently wrong results
  • E.g. always underestimating default risk
  • Minimize noise, beware bias

Correlation is not Causation

  • Important to differentiate when making decisions
  • Causal inference is designed to identify causation
  • Has its own caveats, but better than nothing

Model interpretability

  • Machine learning is great at capturing correlation
  • ML models are often impossible to understand
  • Understanding a model is important to understanding when it will work and when it won’t
  • E.g. ChatGPT seems to be able to “think” until you come across non-existent citations
  • Accountants have to justify their work
  • To auditors, investors, regulators, etc.

Effect Sizes

  • Statistical significance: high certainty in the model
  • Economic significance: how many dollar signs?
  • Remember that models don’t have business sense
  • Important to translate model conclusions into economic impact on costs or revenues

Categorizing Analytical Methods

Categorizing Analytical Methods

Supervised Learning

  • Used when we have outcomes, or “y” variables
  • Approaches model how features map to outcomes
  • Example: predicting types, modeling risk
Supervised Learning

unsupervised Learning

  • Used when we don’t have outcomes
  • Approaches capture structures and relationships
  • Example: identifying groups, computing factors
unsupervised Learning

Healthy Skepticism

  • Analytical models can be very powerful
  • But that doesn’t obviate your professional skepticism
  • Apply your domain knowledge
  • Question the assumptions
  • Question the model
  • Question the data
Analytical Modeling Overview