• Initial analysis to understand data and its structure
• Data types, distributions, statistics
• Can illuminate data issues
• Missing data, outliers, errors
• Identify broad patterns or regularities
• Relationships, clustering, temporal regularity
• Build intuition and inform hypothesis development

• “You can see a lot, just by looking” ~ Yogi Berra
• EDA establishes understanding of the data before assumptions are made or complex modeling techniques are applied
• More EDA now, fewer mistakes later
• Finds “unknown unknowns”
• Be curious and judgmental
• Skepticism is important, look for errors, bias, misinformation, etc.

• Confirmation Bias – the tendency to subconsciously identify, prefer, or interpret information in a way that confirms pre-existing beliefs or hypotheses
• Look for patterns that contradict initial ideas or expectations
• Require higher burden of proof for conclusions that align with priors
• Correlation vs. Causation – relationships discovered with EDA do not necessarily entail causal (functional) relationships
• Observing that two variables move together during EDA does not prove that one causes the other
• Establishing causal relationships requires experiments or causal inference
• Note: correlation is not causation, but they are highly correlated

• “There are 3 kinds of lies: lies, damned lies, and statistics.” ~ Mark Twain
• Visualizations can be misleading (intentionally or not)
• Important to consider whether the interpretation of your visualization will be what is intended

• Visual distortion
• Cherry-picking Data
• Outlier treatment
• Misleading Aggregations
• Comparing Dissimilar Data

• Truncation: Truncating axes can make variation appear greater or less
• Scaling: Scaling axes by non-constant amounts distorts visual comparison
• Log-scale axes are an example of well-understood non-constant scaling
• Twin axes: A second axis can be misleading if using same units as first

• Including only those observations that support a specific conclusion or omitting contradictory evidence
• Removing outliers without justification can present a cleaner, but potentially misleading, picture
• Selecting sub-group may imply the results generalize to the full group

• Using the mean for highly skewed data can be misleading, where the median might be better
• Aggregating data can mask recent changes in trends

• Median is “middle” of data
• Data: https://www.cbo.gov/publication/60706#_idTextAnchor039

• Median is “middle” of data
• Mean (expected value) is weighted by values
• Data: https://www.cbo.gov/publication/60706#_idTextAnchor039

• Median is “middle” of data
• Mean (expected value) is weighted by values
• Be careful to assess what you believe your audience will interpret each to mean
• Data: https://www.cbo.gov/publication/60706#_idTextAnchor039

• Data: https://www.cbo.gov/publication/60706#_idTextAnchor039

• Comparing metrics that aren’t truly comparable (e.g., plotting levels and percentages on the same graph without clear labels)
• Plotting spurious correlations together to imply a relationship
• e.g., tylervigen.com/spurious-correlations

tylervigen.com/spurious-correlations

• Credit Tyler Vigen

• Missing data: drop observations, fill with default, interpolate based on some model
• Input errors: unit errors, transcription errors (OCR, human input)
• Processing errors: conversion logic errors (e.g. date formatting), calculation errors (e.g., divide by 0)
• Duplicates: mistakes or informative
• Outliers: caused by errors, uninformative observations, informative and necessary to understand

• Multiple measurements for single observation
• Journal entry – date, account info, names, amount, etc.
• Consumer – demographics, location, interactions
• Financials
• Multidimensional due to computer representation
• Location stored as latitude and longitude (& height?)
• Text? Images?

• Visualization of three or more dimensions is difficult
• 3D graphs can be hard to interpret without interactivity
• Other features can be introduced (bubble size, color, etc.) but are less self-explanatory and more complex
• Interactive visualizations can help by giving audience control over what they viewing
• Can swap between dimensions being observed
• Can incorporate time / motion

• Overfitting – Insight found during EDA should be understood to be potentially limited to the specific “pull” of the dataset
• Understand conditions of data to determine generalizability
• Multicollinearity – highly correlated variablesidentified by EDA may be problematic in subsequentmodeling or testing

• You are given transaction data for a month
• Date, Amount, Payee, Category, Vendor, Notes, Approver
• Look at data
• Columns, data types, missing values, etc.
• Five-number analysis
• Initial plots
• Line plot of amount over time (Date)
• Bar chart of count by payee, vendor, approver
• Word-cloud of Category and Notes

• Understanding the broader context of the data
• Where did it come from?
• How was it collected?
• Are there known limitations or potential biases in the collection process?
• This is where your accounting knowledge is key

• Understanding the dataset at hand
• What do the variables represent?
• What contextual information should be incorporated into understanding this data?
• Is more data needed before use (merging)?

• Examine the dataset’s dimensions (# of rows and columns)
• Identify the data types of each variable
• numerical continuous/discrete, categorical nominal/ordinal, date/time, text short/long
• View rows of data to get a sense of the content and structure
• Identify “special” columns like merge keys, unique IDs
• Identify how missing observations are represented
• Identify whether variables have default values
• Investigate whether and how missings can be filled/imputed
• Identify whether duplicates exist and can be removed

• “There is no excuse for failing to plot and look” ~ John Tukey
• Histograms: Understand the distribution of numerical variables, identify outliers
• Box Plots: Visualize the distribution, central tendency, spread, and outliers of numerical data, possibly by category
• Bar Charts: Compare categorical data
• Scatter Plots: Investigate relationships between numerical variables
• Line Charts: Show trends over time, across categories
• Heatmaps: Visualize correlation matrices, missing values, etc.

• Start looking for abnormalities & outliers
• Amounts by day of week (box plot)
• Amounts over time by approver (line plot)
• Distribution of amounts by approver (box plot)
• Amounts by vendor (histogram)

• Groups up to 5 optional
• Deliverable pdf slide-deck