How do you choose a classification threshold?

Context

You built a binary sentiment classification model (e.g., positive vs. negative) and need to deploy it in a product where actions depend on the model’s output.

Questions

1. Walk through your ML pipeline end-to-end:
   • Data sourcing/labeling and dataset construction (train/validation/test splits).
   • Feature design or model choice (e.g., TF-IDF + linear model vs. transformer).
   • Training procedure and evaluation setup.
   • Key practical challenges (class imbalance, noisy labels, distribution shift) and how you handled them.
2. Modeling choices:
   • Why did you choose method/model X over alternatives?
   • What assumptions does it make, and what trade-offs does it introduce (latency, interpretability, cost, robustness)?
3. Iterative refinement:
   • Describe how you iteratively improved the system (e.g., error analysis → new features/data → retrain → re-evaluate).
   • What were your biggest learnings and what would you do differently next time?
4. Threshold selection for deployment:
   • Your model outputs a probability score. How do you choose the decision threshold ?
   • Which metrics would you consider (precision, recall, F1, ROC-AUC, PR-AUC, cost-weighted loss), and which would be primary vs. diagnostic vs. guardrail ?
   • How would the answer change under:
     • Severe class imbalance
     • Different costs for false positives vs. false negatives
     • A fixed review/ops capacity (e.g., only 1,000 items/day can be escalated)
5. Metric definition:
   • If a stakeholder proposes defining “success” as metric XXX , how do you evaluate whether that definition is appropriate?
   • What data issues (label leakage, delayed labels, sampling bias) could make the metric misleading?