# Model Answers — Behavioral & Technical Prompts (Data Scientist) These four prompts test different competencies, so use four distinct stories. The patterns below work for any candidate; **swap in your own real examples and numbers** — interviewers in financial services will probe the details, so never present figures you cannot defend. ## How to approach every answer - **STAR with discipline:** state the Situation/Task in one or two sentences, then spend most of your time on *your* Actions (first-person — "I", not "we"), and close with a quantified Result. Add a one-line Reflection where it shows growth. - **Quantify and connect to business value:** percent change, dollars, time saved, users/transactions affected. Always translate a model metric into something the org cares about ("28% fewer false positives → fewer customers wrongly declined → CSAT +3 pts"). - **Make trade-offs explicit:** name the alternative you rejected and why. Judgment is the signal, not the buzzword. - **Show rigor and risk awareness** (especially for regulated financial data): leakage checks, time-aware validation, calibration, explainability, fairness, and guardrails. --- ## Part 1 — Initiative and Ownership **Situation.** Product teams were running dozens of A/B tests, but ~40-50% were underpowered, producing inconclusive results and wasted release cycles. Nobody owned experiment design — analysts copied old configs. **Task.** I was not asked to fix this, but I saw a recurring, cross-team pain point and decided to make rigorous experiment design self-serve. **Action (what I did beyond scope).** - Built a self-serve experiment-design app (Python + Streamlit) with calculators for proportions and means, CUPED variance reduction, and minimum-detectable-effect (MDE) guidance. - Implemented the sample-size formula for a two-proportion test, $n = \dfrac{2\,(z_{1-\alpha/2} + z_{1-\beta})^2\, p(1-p)}{\Delta^2}$, with sensible defaults for baseline $p$ and a business-relevant $\Delta$. - Added guardrails: a sample-ratio-mismatch (SRM) chi-square check, sequential-look guidance, and a pre-launch checklist (primary metric, segmentation, pre-registration, power). - Ran three workshops and wrote a two-page playbook with worked examples. **Result.** - Share of adequately powered tests rose from **52% → 86%** over two quarters. - Average test duration fell **~18%** (CUPED + better MDE setting). - Saved an estimated **25-30 analyst hours/month** and meaningfully improved trust in the experimentation program. **Why it shows initiative:** I identified the problem, built and shipped a tool, trained users, and measured the impact — none of which was assigned. --- ## Part 2 — Favorite Quantitative/Technical Project **End goal & business context.** Reduce **false positives** (legitimate transactions wrongly flagged) in a fraud-screening system while holding fraud recall constant. False positives create customer friction and review load; missed fraud creates direct loss — an asymmetric-cost problem. **Tools and why (each justified vs. an alternative).** - **SQL (Snowflake)** for feature extraction and time-based joins — scalable warehouse with strong window functions; chosen over in-Python aggregation because the data volume made warehouse-side computation far cheaper. - **Python (pandas, scikit-learn, LightGBM)** — gradient-boosted trees handle heterogeneous tabular features, nonlinear interactions, and missingness natively; chosen over a deep net because the data was tabular and explainability mattered. - **MLflow** for experiment tracking, **SHAP** for explainability (required by Risk), **Great Expectations** for data-quality checks, **Airflow** for batch scoring. **Methods.** - **Leakage prevention:** used only *pre-authorization* features; any signal computed after the transaction outcome was excluded, and the pipeline was time-aware. - **Feature engineering:** rolling aggregates (transaction count/amount by card, device, merchant over 1h/24h/7d), velocity features, device-merchant affinity, geodistance between consecutive transactions. - **Validation:** time-based splits (fold by month) to respect temporal dependence — a random split here would leak the future into the past. - **Imbalance:** class-weighted loss; monitored PR-AUC and recall at a fixed false-positive rate rather than ROC-AUC alone. - **Baseline → model:** logistic-regression baseline, then LightGBM tuned with Bayesian optimization; **calibrated** probabilities (isotonic) and tuned the threshold for the asymmetric cost. - **Objective:** minimize expected cost $\,\mathbb{E}[\text{cost}] = c_{FP}\cdot FP + c_{FN}\cdot FN\,$ subject to $\text{recall} \ge \text{target}$. **Worked numeric example.** For a representative month of 100,000 transactions with 200 fraud (0.2%): - Baseline at 95% recall, 0.50% FP rate → $FP \approx 0.005 \times 99{,}800 \approx 499$, $FN = 10$. - Tuned LightGBM at the **same** 95% recall, 0.36% FP rate → $FP \approx 359$, $FN = 10$. - With $c_{FP}=\$5$ (customer friction) and $c_{FN}=\$500$ (loss): baseline cost $\approx 499{\times}5 + 10{\times}500 = \$7{,}495$; new cost $\approx 359{\times}5 + 10{\times}500 = \$6{,}795$ → **~$700 saved per month-sample**, which scaled to over **$1.2M/year** at production volume. **Obstacles & mitigations.** - *Leakage:* enforced time-aware pipelines and excluded post-authorization features. - *Imbalance:* class weights plus focal-loss experiments; tracked PR-AUC and recall@fixed-FPR. - *Drift:* PSI/KS monitoring on key features and calibration-drift checks, with alerts that triggered retraining. **Implementation.** - Deployed a batch-scoring Airflow DAG with canary routing (10% traffic) and a kill switch. - Logged SHAP top features as **reason codes** per decision for Risk review. - Validated with SRM checks, pre-registered metrics (recall@fixed-FPR, customer-contact rate), and a two-week holdout. **Outcome.** False positives **−28%** at constant 95% recall; customer-contact rate **−22%**; CSAT **+3.1 pts**; analyst review load **−18%**. **Pitfalls I call out:** random splits causing temporal leakage; relying on ROC-AUC for rare events instead of PR-AUC/cost-sensitive metrics; deploying without checking calibration curves before thresholding. --- ## Part 3 — Complex Problem Solving **Problem.** An onboarding UI change "won" an A/B test on conversion (**+2.4 pts**), yet downstream revenue and activations didn't move. Stakeholders questioned the measurement. **Analysis.** I suspected a **mix-shift / Simpson's paradox** rather than a real effect, so I segmented by acquisition channel and device. Illustrative numbers: - **Control:** Paid 30k @ 8% (2,400) + Organic 70k @ 2% (1,400) → overall **3.8%** (3,800/100k). - **Treatment:** Paid 70k @ 8% (5,600) + Organic 30k @ 2% (600) → overall **6.2%** (6,200/100k). - **Within every stratum the treatment effect was ~0** — the aggregate "lift" came entirely from treatment receiving more high-converting Paid traffic, a sign of broken randomization (SRM). **Solution & rationale.** - Reanalyzed with **stratified estimates** (Cochran-Mantel-Haenszel weighting) matched to historical channel proportions → net effect ≈ 0. I chose stratification over simply re-running because it both diagnosed *and* corrected the bias from existing data. - Fixed the root cause: implemented **stratified randomization** (randomization key = `user_id × channel bucket`) and added reweighting in the analysis pipeline. - Added an **SRM chi-square alert** ($p < 0.01$) and pre-registered segmentation. **Obstacles & trade-offs.** - *Low-sample strata:* merged adjacent device strata with similar baselines and used variance-stabilizing transforms; documented a minimum-stratum-size rule. - *Organizational pushback* from teams invested in the "win": addressed with a neutral postmortem walking through the math, then a clean rerun under stratified allocation. **Result.** The rerun showed a **0.1 pt (non-significant)** effect — we avoided shipping a non-impactful change to millions of users. The new guardrails cut SRM incidents **~80%** the following quarter and raised leadership confidence in the experimentation platform. **Reflection.** Complex problems usually hinge on **identification assumptions** (balanced allocation, no confounding). Stratification and SRM checks are low-cost, high-leverage safeguards. --- ## Part 4 — Relationship Building **Situation.** A senior Risk Manager was skeptical of ML-driven decisions on explainability and compliance grounds and was effectively blocking model adoption. **Actions (specific, to build trust).** - Set up recurring 1:1s to understand their risk appetite, required disclosures, and audit-trail needs *before* defending the model. - **Co-designed acceptance criteria** with them: confusion-matrix thresholds, reason-code coverage, and fairness checks across protected groups — so the bar was jointly owned. - Built a "risk lens" dashboard: for any threshold, show expected FP/FN counts, cost curves, and top SHAP reason codes. - Ran a **champion-challenger pilot** with a small exposure cap and daily review, letting trust accrue from evidence rather than promises. **Challenges & how I addressed them.** - *Different vocabularies:* mapped model metrics to risk terms (recall → "coverage", precision → "purity") and added glossary tooltips. - *Their time constraints:* sent one-page briefs before meetings and async recorded walkthroughs to cut meeting load. **Outcome.** The model was approved for broader rollout with **jointly owned thresholds**; per-case review time **−30%**; false positives **−18%** with no recall loss. We turned the acceptance criteria into a reusable template that shortened later approval cycles **~25%**. **Reflection.** Trust came from aligning on the other person's incentives, making trade-offs explicit, and creating shared artifacts (dashboards, criteria) that removed ambiguity. --- ## Addressing the follow-up questions - **Ship the fraud model with half the data or tight latency:** with half the data I'd lean on stronger regularization and simpler features, fall back to the logistic baseline where the boosted model is unstable, and widen confidence intervals before committing thresholds. Under a tight latency budget I'd precompute the rolling/velocity features in the warehouse and serve a slimmer feature set, or distill the model, accepting a small recall trade-off for in-SLA scoring. - **Convincing a stakeholder invested in the wrong conclusion:** I didn't argue the conclusion — I rebuilt trust in the *method*. A neutral, numbers-first postmortem showing the within-stratum effects, plus a transparent rerun under corrected randomization, let them reach the new conclusion themselves rather than being told they were wrong. - **A time my initiative failed:** an earlier "automated insights" digest I built saw near-zero adoption because I optimized for cleverness, not for a question users actually had. Lesson: validate demand with a few users before building. I now ship a manual prototype first and only automate once it's pulled, not pushed. - **Sustaining trust after the win:** I kept the Risk Manager on the recurring review and the shared dashboard, so drift or a fairness regression surfaced to *both* of us. It was tested when a feature drifted and false positives crept up — because the monitoring was jointly owned, we caught and retrained it together instead of it becoming an adversarial escalation. --- ## Reusable summary playbook - **Frame** every story with STAR; quantify impact and tie it to business outcomes. - **For models:** prevent leakage, use time-aware validation, prefer cost-sensitive metrics for rare events, calibrate before thresholding, and provide explainability. - **For experiments:** plan power/MDE, check SRM, pre-register metrics and segments, and stratify when allocation can drift. - **For relationships:** align on the other side's KPIs, make trade-offs transparent, and build trust through low-risk pilots and shared artifacts.