Linear Regression, OLS Diagnostics And Assumptions

What's being tested

Interviewers probe whether you can build, validate, and defend a linear regression model for inference and prediction: know the core Ordinary Least Squares (OLS) mechanics, the assumptions that justify unbiased/efficient estimates, how to detect common violations with diagnostics, and what corrective actions preserve interpretability. Amazon cares because DS work often requires causal reasoning, trustworthy A/B metric adjustment, and clear effect-size communication under real-world messiness.

Core knowledge

OLS estimator: β̂ = (X'X)^{-1}X'y; under Gauss‑Markov assumptions β̂ is the BLUE (best linear unbiased estimator). Var(β̂) = σ^2 (X'X)^{-1}.
Key assumptions: linearity in parameters, exogeneity E[ε|X]=0, homoskedasticity Var(ε|X)=σ^2, no perfect multicollinearity, independence of errors, and correct functional form. Violation of exogeneity causes bias; others affect efficiency and inference.
Residual diagnostics: plot residuals vs fitted for nonlinearity/heteroskedasticity, QQ-plot for normality of errors (only crucial for small-sample t/CIs), and partial-residual plots to check single-predictor nonlinearity.
Heteroskedasticity tests & fixes: run Breusch-Pagan or White test (p<0.05 indicates heteroskedasticity); use heteroskedasticity-consistent SEs (HC0–HC3) or transform outcome (log) / model variance (GLM).
Autocorrelation: for time series use Durbin-Watson (≈2 ok; <<2 positive autocorr), or include AR terms / use Newey‑West SEs for serial correlation.
Multicollinearity: measure with VIF; VIF>10 (or >5 as conservative) indicates problematic collinearity. Remedies: drop/merge features, principal components, or use ridge (introduces bias; not for causal coefficient interpretation).
Influence & outliers: compute hat matrix leverage h_ii (avg p/n); rule: high leverage > 2p/n, Cook's distance > 4/n flags influential points; inspect and justify any removal.
Inference mechanics: t-stat = β̂ / SE(β̂), F-test for joint hypotheses; report effect sizes and 95% CIs alongside p-values; adjusted R^2 penalizes for number of regressors.
Endogeneity & omitted variable bias: if X correlated with ε, OLS is biased (direction from omitted confounder correlates with both). High-level remedy: 2SLS / instrumental variables or design-based identification (randomization).
Measurement error: classical error-in-X attenuates coefficients toward zero; address via validation data or IV.
Sampling & clustering: when observations cluster (users, regions), use cluster-robust SEs; need many clusters (rule-of-thumb > ~50); for few clusters use wild cluster bootstrap.
Practical sample-size guidance: for stable SEs and asymptotic normality, aim for n >> p (p predictors), ideally n/p > 10–20; for subgroup inference or cluster SEs need larger n per group.
When to favor prediction vs inference: use cross-validation and regularized models (LASSO, XGBoost) for predictive accuracy; for causal estimates prefer specification that preserves interpretability and valid SEs.

Worked example

Scenario: predict weekly purchase amount from ad_spend, price, and seasonality, and diagnostics show heteroskedastic residuals and high VIF between ad_spend and price. First 30 seconds: confirm objective (prediction or causal effect of ad_spend?), data scope (aggregated weekly, number of weeks, clustering by region?), and variable construction (lags, interactions). Skeleton of a strong answer: (1) state assumption violations and their consequences (heteroskedasticity → invalid SEs; multicollinearity → inflated SEs, unstable coefficients), (2) run diagnostics (residuals vs fitted, Breusch-Pagan, VIF, Cook's distance), (3) propose fixes (use HC3 robust SEs for inference; if collinearity prevents interpreting ad_spend vs price, consider combining into marketing intensity, orthogonalize via residualization, or apply ridge if prediction prioritized), (4) re-evaluate and report effect sizes with CI and sensitivity checks. Tradeoff flagged: using ridge improves prediction but shrinks coefficients—unsuitable if you must report unbiased causal effect. Close by saying: "if more time, I'd attempt an IV for ad_spend (budget shock) or run subgroup/stability analyses and bootstrap SEs."

A second angle

Consider a time-series regression estimating daily DAU using past DAU, promotion flags, and feature release indicators. Same OLS mechanics apply, but key constraints shift: autocorrelation and nonstationarity matter more than cross-sectional collinearity. The candidate should prioritize checking stationarity (ADF test), include lagged dependent variable or use difference-in-differences if appropriate, and compute Newey‑West or model AR errors. If promotions are endogenously timed with demand, discuss identification (instrument, natural experiment) rather than just adding controls. This reframing tests both diagnostic fluency and causal thinking under temporal dependence.

Common pitfalls

Pitfall: Interpreting robust SEs as fixing endogeneity — robust/clustered SEs only adjust inference for variance misspecification or dependence; they do not remove bias from omitted/confounded regressors.

Pitfall: Dropping predictors purely to reduce VIF without considering omitted-variable bias — removing a confounded but important control can introduce bias in target coefficients.

Pitfall: Over-reliance on R^2 for model quality — a high R^2 doesn’t imply causal identification or correct specification; always examine residual structure and substantive plausibility.

Connections

This topic commonly leads into instrumental variables / 2SLS for endogeneity, generalized linear models for non-normal outcomes, and mixed-effects / hierarchical models when observations are nested or clustered.