Model Calibration And Evaluation

What's being tested

Candidates must show practical mastery of probability calibration and model evaluation for classifiers and rankers: how to measure calibrated probabilities, fix miscalibration, choose and justify evaluation metrics, and reason about offline→online gaps. LinkedIn cares because downstream systems (ranking, thresholding, personalization) rely on well-calibrated scores and appropriate evaluation to make safe, business-aligned decisions.

Core knowledge

• Calibration: the property that reported probability p means the event occurs ≈p fraction of the time; essential when scores drive thresholds, costs, or downstream probabilistic models.

• Reliability diagram: plot observed accuracy vs predicted confidence across bins; use equal-width or equal-frequency bins and show sample counts per bin to avoid misleading visuals.

• Expected Calibration Error (ECE): common scalar summary: $\text{ECE}=\sum_{m=1}^M\frac{|B_m|}{n}\big|\text{acc}(B_m)-\text{conf}(B_m)\big|$ where bins B_m partition examples; sensitive to binning choice and class imbalance.

• Brier score and log-loss: Brier = $\frac{1}{n}\sum_{i}(p_i-y_i)^2$; log-loss penalizes confident wrong predictions more; both combine calibration + discrimination.

• Discrimination metrics: AUC-ROC, AUC-PR (prefer PR under heavy class imbalance), precision@k, NDCG, MRR for ranking — pick metrics aligned to task utility (e.g., top-k recall for candidate generation).

• Post-hoc calibration methods: Platt scaling (logistic regression on scores), isotonic regression (monotonic non-parametric mapping; risk of overfitting), temperature scaling (single scalar T on logits; minimal params, good for modern nets). Use a held-out validation/calibration set, never the training set.

• Multi-class calibration: apply temperature or vector/matrix scaling to logits; can use one-vs-rest Platt for large label sets; measure per-class ECE or class-weighted ECE.

• Data shift and recalibration: calibration can break under covariate or label shift. Monitor distribution drift and recalibrate periodically; prefer fast methods like temperature scaling for online re-calibration.

• Practical tooling: sklearn.calibration.CalibratedClassifierCV for Platt/isotonic; scikit-learn, PyTorch/TensorFlow for logits and temperature scaling; Prometheus/Grafana or custom metrics to track calibration drift in production.

• Evaluation design: use time-based splits for temporal data, stratified sampling for rare classes, and cross-validation for small datasets; compute confidence intervals via bootstrapping for metrics like ECE or AUC.

• Threshold selection: determine thresholds by optimizing task-specific utility (F1, cost-weighted loss, business KPIs) on validation data and test robustness under shifted distributions.

• Ranking vs scoring: when converting ranking scores to probabilities, validate calibration against implicit feedback biases (exposure, position); consider counterfactual evaluation (IPS) if evaluating causal effects.

Worked example — "Answer practical ML foundations questions"

Frame: ask which downstream decisions use the probabilities, class imbalance severity, and whether calibration must hold across cohorts (geography, device). Declare assumptions: offline labeled validation set exists and distribution approximates production. Structure answer into three pillars: (1) diagnostic — compute reliability diagram, ECE, Brier, and AUC-PR; (2) remediation — try temperature scaling first (low-risk), then Platt or isotonic regression if non-monotonic errors persist; (3) deployment & monitoring — validate on held-out set, push to canary traffic, track online calibration metrics and trigger recalibration. A key tradeoff: isotonic regression can overfit with small calibration sets but models arbitrary monotone distortions; temperature scaling is robust with small sets but only fixes "softness" of logits. Close by saying: "If I had more time I'd run per-cohort ECE, test matrix-scaling for large multiclass error, and simulate label shift to stress-test recalibration frequency."

A second angle — "Design LinkedIn Learning course recommendations"

Here the same calibration and evaluation concepts focus on ranking and conversion modeling. You'd predict click/engagement probabilities for personalized ranking; require calibrated scores for fair exploration and downstream decision policy (e.g., allocate impressions). Diagnostics: compute calibration conditional on position/exposure to control for position bias, and evaluate NDCG or precision@k alongside ECE. For remediation, fit propensity-corrected calibration (re-weight examples by exposure IPS) or calibrate on logged-exposure data. Operational constraints (latency, model size) push toward lightweight calibrators like temperature scaling served with the model, and monitoring must detect shifts in content tastes so you can retrain and recalibrate pipelines.

Common pitfalls

Pitfall: Reporting low ECE as "good" without checking class imbalance.

ECE can be small when most mass is on a well-calibrated majority class; compute per-class or class-weighted ECE and examine reliability diagrams.

Pitfall: Calibrating on the training set or using the test set for tuning.

Always use a separate calibration/validation set to fit Platt/isotonic/temperature parameters; otherwise calibration will be overoptimistic.

Pitfall: Equating high AUC with good probabilities.

A model with excellent ranking (AUC) can be badly calibrated; decide early whether ranking or well-calibrated probabilities drive product decisions and optimize accordingly.

Connections

Interviewers often pivot to uncertainty quantification (Bayesian nets, MC-dropout), online monitoring & drift detection (dataset shift alarms, retraining triggers), or experiment design (A/B test sensitivity when using thresholds). Be prepared to connect calibration fixes to deployment mechanics and monitoring.

Further reading

• On Calibration of Modern Neural Networks — Guo et al., 2017 — seminal paper showing temperature scaling effectiveness.

• Predicting Good Probabilities — Niculescu-Mizil & Caruana, 2005 — empirical comparison of calibration methods.