Evaluate a Binary Model with Confusion Counts, Calibration, and Normalized Entropy
Company: Snapchat
Role: Machine Learning Engineer
Category: Machine Learning
Difficulty: medium
Interview Round: Onsite
Explain the four cells of a binary classification confusion matrix, useful offline metrics, model calibration, and normalized entropy. Connect each measure to what it can and cannot establish about a model.
### Constraints and Clarifications
Use a binary target y in{0,1} and a predicted probability p. State the decision threshold when discussing confusion counts. For normalized entropy, use average binary log loss divided by the entropy of the evaluation set's positive prevalence; this convention must be stated because metric names can vary.
### What a Strong Answer Covers
TP,FP,TN,FN; precision and recall; threshold-independent probability scoring; calibration; the normalized-entropy formula and its degenerate cases. Do not infer an employer's internal metric definition from the name alone.
### Follow-up Questions
- Can a model rank examples well but have poorly calibrated probabilities?
- What happens to normalized entropy when the evaluation set contains only one class?
Overview: Explain confusion-matrix cells, probability calibration, and normalized entropy in one evaluation framework. Define the metric convention, handle degenerate prevalence, and distinguish ranking from calibrated prediction.
Read the full Snapchat Machine Learning Engineer interview experience this question came from