System Design: End-to-End Email Spam Detection

Context

Design an end-to-end system that detects and handles spam emails at scale. Assume you are building for a large consumer email service handling high throughput and strict latency requirements. The design should cover data, ML, serving, experimentation, and operations.

Requirements

1. Problem Definition and Labeling
   • Define the objective(s) and action outcomes (e.g., block, quarantine, inbox with banner).
   • Labeling sources and policies.
2. Data Sources and Collection
   • Inbound traffic, user reports, honeypots, abuse teams, reputation feeds.
   • Collection, sampling, retention, and governance.
3. Feature Engineering
   • Content features (text, URLs, attachments), headers, sender/domain/IP reputation, network/behavioral signals.
4. Model Choices and Training
   • Baseline rules, supervised ML models, online learning.
   • Handling class imbalance, feature hashing, model calibration.
5. Serving Architecture and Constraints
   • Placement in the mail pipeline, APIs, latency/throughput targets, caching, fallbacks.
6. Thresholding and Calibration
   • Score-to-action mapping, per-segment thresholds, calibration methods.
7. Evaluation Metrics
   • Precision, recall, ROC/PR analysis, and cost-weighted metrics.
8. Abuse/Adversarial Defenses and Feedback Loops
   • Evasion tactics, spoofing defenses, URL/attachment handling, user feedback integration.
9. Cold Start, Concept Drift, Retraining Cadence
   • New senders/domains, seasonal drift, automated retraining.
10. Online Experimentation
    • A/B testing, ramp strategies, guardrails.
11. Monitoring, Logging, Rollback
    • Real-time and batch monitoring, alerting, safe rollback.
12. Privacy and Compliance
    • Data minimization, encryption, regional residency, user controls.