Choose Evaluation Metrics and a Decision Threshold for a Binary Classifier

During a resume deep dive on a project in which you trained a binary classifier, the interviewer asks how you evaluated the model and, because the application needs a yes/no decision rather than a score, how you would set the classification threshold. In the reported exchange, an answer about comparing against baselines and normalizing, shuffling and balancing the data did not satisfy the interviewer, who said the threshold has to be chosen for the specific problem. Answer for a binary classifier whose use case you can describe, and ask about anything the interviewer leaves open. ### Clarifying Questions - What decision does the classifier drive, and what happens after a positive prediction: an automatic action, or a case sent to a person for review? - What does a false positive cost compared with a false negative? - How rare is the positive class, in the training data and in production? - Is there a limit on how many positives can be acted on per day, or a required minimum recall or precision? - Are the model's scores calibrated probabilities, and was the training data resampled or reweighted? ### Part 1 — Choose evaluation metrics Explain which metrics you would report for the classifier, what each one tells you, and which one you would use to compare candidate models. ```hint Rare positives Suppose only a small fraction of cases are positive. Ask which metrics would still look good for a model that misses most of them. ``` #### What This Part Should Cover - Threshold-dependent metrics derived from the confusion matrix, and when each one matters - Threshold-free ranking metrics, and how they behave when positives are rare - Whether the scores can be trusted as probabilities, and a metric tied to the use case ### Part 2 — Set the decision threshold Explain how you would choose the threshold that turns scores into decisions, how you would validate it, and how you would keep it right after launch. ```hint What a mistake costs Ask what happens downstream when the model is wrong in each direction, and whether those two mistakes cost the same. ``` ```hint Where to tune it Decide which data split the threshold is chosen on, and what changes if the class mix in production differs from the data the model was trained on. ``` #### What This Part Should Cover - Linking the threshold to the costs of errors or to an operating constraint such as review capacity or a minimum recall - Choosing and validating the threshold without contaminating the final test estimate - How resampling, calibration and shifts in the positive rate affect the chosen threshold ### What a Strong Answer Covers - Metrics chosen for the use case rather than listed by rote - A clear distinction between ranking quality and the decision threshold - A threshold rule derived from costs or constraints, illustrated with a worked example - Awareness of class imbalance, calibration and prior shift - Monitoring and re-tuning the threshold after launch ### Follow-up Questions - If positive cases are very rare, why can ROC AUC look excellent while precision at a useful threshold is poor? - You trained on a dataset downsampled to equal class sizes. How must the scores and the threshold change before deployment? - A review team can handle only a fixed number of flagged cases per day. How does that change how you pick the threshold? - Would you ever use more than one threshold?