# Diagnose a Production Classifier and Defend Its Systems Design You own a binary classifier used by a downstream production service. The positive class is rare, the operating team reports a recent accuracy drop, and the service must record cumulative rocket-engine flight time for later analysis. ### Constraints & Assumptions - Training labels can arrive late and may be noisier for one class. - The decision threshold can be changed independently of model retraining. - Flight-time events can be duplicated or delivered out of order. - Give designs and diagnostics rather than company-specific infrastructure claims. ### Clarifying Questions to Ask - What business cost is attached to a false positive and a false negative? - Is the reported accuracy drop based on online labels, delayed labels, or a proxy? - Does engine flight time arrive as intervals, cumulative counters, or start/stop events? ### Part 1 — Model selection and thresholding Explain how you would tune hyperparameters, select metrics, and choose a precision-recall operating point for an imbalanced dataset. #### What This Part Should Cover - A leakage-safe validation plan - Metric and threshold choices tied to error costs - Class weighting, resampling, or calibrated-probability trade-offs ### Part 2 — Production validation and debugging Describe how the output should be validated downstream and how you would investigate the accuracy drop without immediately retraining. #### What This Part Should Cover - Data, label, feature, and concept-drift checks - Slice analysis and training-serving skew - A rollback or shadow-evaluation plan ### Part 3 — Flight-time data model Design an idempotent event model and an aggregation method for cumulative engine flight time when events may be late, duplicated, or corrected. #### What This Part Should Cover - Stable event identity and units - Correction and overlap semantics - Recomputation and auditability ### Part 4 — Implementation trade-offs Compare Python with C++ for the online component, then explain when threads, processes, stack allocation, and heap allocation matter. #### What This Part Should Cover - GIL and native-extension nuance - Isolation versus shared-memory costs - Lifetime, ownership, and resource-bound reasoning ### What a Strong Answer Covers - A disciplined order of operations - Explicit assumptions and invariants - Failure containment and observable evidence ```hint Separate the failure planes Treat model quality, serving correctness, label quality, and event-accounting correctness as separate hypotheses. A single dashboard metric cannot identify which plane failed. ``` ### Follow-up Questions - How would you detect that the chosen threshold has become stale? - How would you backfill corrected flight-time events without double counting?