Design an end-to-end spam detection system
Overview
This question evaluates a data scientist's system-design and applied machine learning engineering skills—covering problem framing and labeling, feature representation, model selection and calibration, real-time serving constraints, drift detection, and feedback/safety mechanisms—and is commonly asked to probe trade-offs between latency, precision/recall, and robustness against adversarial evolution in production spam detection. Category: Machine Learning; it tests machine learning systems and production-ML competencies at both conceptual-design and practical-application levels, emphasizing calibration, evaluation (offline and online), operational reliability, and rollback/mitigation planning.
Design an End-to-End Email Spam Detection System
You are asked to design a production-grade email spam detection system that meets the following constraints:
- Real-time scoring with p99 latency < 50 ms.
- Minimize false positives (target precision ≥ 98% for hard blocks), while keeping recall high.
- Adversaries evolve tactics continuously.
Address the following:
-
Problem Framing and Labeling
- Define classes: ham vs spam, and graymail (legitimate but unwanted marketing/notifications).
- Discuss labeling sources and strategy, including handling noisy/weak labels and delayed abuse reports.
-
Features and Representations
- Propose key features: character/word n-grams, sender/domain/IP reputation, URL features, MIME structure, lightweight embeddings.
- Explain how to prevent data leakage (e.g., future knowledge, reply/forward chains, time-based leakage).
-
Model Choice and Serving
- Compare models (logistic regression, gradient boosting, compact transformer) given latency and adversarial drift.
- Describe calibration and thresholding for different enforcement actions (block, quarantine, tag).
-
Training Pipeline, Sampling, and Drift Detection
- Outline the end-to-end training pipeline and sampling to handle class imbalance.
- Describe drift detection (population/stability metrics, canaries) and retraining triggers.
-
Evaluation
- Offline: metrics such as PR-AUC; calibrated precision/recall at business thresholds.
- Online: A/B design, guardrails, and holdouts.
-
Feedback Loops and Safety
- Appeals workflow, human-in-the-loop review.
- Bias, privacy, and PII handling.
-
Cost, Reliability, and Rollback
- Compute/latency budget, reliability/SLOs, and rollback plans.
Finally, list the top three failure modes you anticipate and concrete mitigations for each.
Solution
Login required