Demonstrate JD skills with quantified outcomes

# Sample, teaching-oriented answer Chosen JD-highlighted skill: Experimentation and causal inference (A/B testing, metric design) Resume project: Personalization experiment to improve the homepage ranking strategy at a large subscription streaming platform. ## 1) Problem, constraints, success metric - Problem: Increase content discovery from the homepage without harming quality-of-experience (QoE). - Constraints: - Latency: p95 homepage render + ranking budget < 150 ms. - Global rollout across regions/languages and device types (TV, mobile, web). - Guardrails: No meaningful increase in rebuffering/error rates; no policy/rights violations. - Experiment overlap policy: Mutually exclusive buckets with other homepage tests. - Primary success metric: - 7-day Play Starts per Profile (PSP). Secondary: 7-day Watch Time per Profile (minutes). Guardrails: Start-failure rate, Rebuffering ratio, Crash rate. - MDE/power target: - Detect a 1.0% relative lift on PSP with 80% power, α=0.05. - Two-sample size approximation per arm: n ≈ 2 · (z_{1−α/2}+z_{1−β})^2 · σ^2 / δ^2. - Example inputs: baseline mean μ=2.9 plays, σ=3.2, δ=0.029 (1% of μ), z_{0.975}=1.96, z_{0.8}=0.84. - n ≈ 2·(1.96+0.84)^2·(3.2^2)/(0.029^2) ≈ ~191k profiles/arm (pre-variance-reduction). We expected to hit this in <1 day. ## 2) Techniques, tools, and non-obvious design choices - Experiment design: - Randomization unit: Profile-level; cluster-robust analysis at household level to mitigate cross-device interference. - 50/50 allocation, stratified by region × device to improve balance and power. - Variance reduction: CUPED using 14-day pre-experiment covariates (prior play starts, watch time, tenure, device). - CUPED formula: Y_adj = Y − θ·(X − E[X]), where θ = Cov(Y, X) / Var(X). - Analysis: - Primary estimator: Difference-in-means on per-profile outcomes; confirmatory OLS with covariates and cluster-robust SEs (household clustering). - Ratio metrics handled via per-profile aggregation (avoid per-event ratios) and delta-method checks; confirm via nonparametric bootstrap (10k reps). - Sequential monitoring with alpha-spending (Pocock boundary) to avoid inflated Type I error during gated ramps. - Ranking/modeling: - Offline reranking blend: baseline collaborative filtering + short-term session signals; limited to top-N candidates to stay within latency. - Non-obvious choice: Winsorized extreme watch-time at 99.5% to stabilize variance; capped per-request reranking to 50 candidates to fit p95 latency. - Tooling and scale: - Data/compute: PySpark 3.3 on Spark 3.3 (Databricks Runtime 12.x), Delta tables. - Orchestration: Airflow 2.6 for daily ETL and metric rollups; MLflow 2.6 for experiment metadata. - Stats: Python 3.10, statsmodels 0.14, SciPy 1.10; visualization in a BI tool for stakeholder readouts. - Scale: ~12M profiles in experiment over 14 days; ~2B events/day feeding metrics. ## 3) Nontrivial failure/edge case and resolution - Issue: Sample Ratio Mismatch (SRM) on Android WebView (51.3/48.7 split, p<1e−4). Root cause was CDN-level caching of the pre-assigned homepage for some anonymous sessions before server-side assignment was finalized. - Resolution: - Moved assignment to server-side earlier in the request pipeline; used a stable profile_id-based Murmur3 hash for bucketing. - Added real-time SRM monitoring (hourly Pearson χ² across key strata) and blocked enrollment when SRM triggered. - Post-fix, arm proportions were within ±0.1% of expected across strata; we invalidated pre-fix data and restarted the experiment. - Lesson: For pages served behind aggressive edge caches, ensure treatment assignment occurs upstream of any cacheable content and that anonymous flows get a stable assignment key. ## 4) Impact (before/after) and what I’d change next time - Results (14 days, after fix, CUPED-adjusted): - +1.8% relative lift in 7-day Play Starts per Profile (ATE +0.052 from 2.90 baseline), 95% CI [+1.0%, +2.6%], p=0.001. - +1.2% lift in 7-day Watch Time per Profile (≈ +4.1 minutes), 95% CI [+1.4, +6.8] minutes. - Guardrails: Rebuffering +0.03pp (ns), Start-failure −0.02pp (ns). No material QoE regressions. - Heterogeneity: Larger lift on new users (<30 days tenure): +3.4% PSP; stable for long-tenure users. - Business translation: - At full rollout scale, the lift implies several million incremental weekly play starts with stable QoE. - If doing it again: - Pre-register stratified MDEs and power by user tenure to right-size ramp windows. - Add CUPAC (covariate-assisted randomization) to further reduce variance and speed decisions. - Use a short, pre-launch shadow test with off-policy evaluation (doubly robust estimator) to catch SRM-like issues before live ramp. ## 5) De-risking with stakeholders and conscious trade-offs - De-risking steps: - Alignment on primary/guardrail metrics and decision thresholds before launch; documented in a one-pager and pre-registered. - Gated rollout: 1% → 5% → 20% → 50% with alpha-spent interim looks and automatic rollback on QoE guardrail breaches. - Data-quality checks in Airflow using Great Expectations (schema, nulls, range checks) and automated SRM alerts. - Mutually exclusive bucketing with other homepage experiments to avoid interference. - Trade-offs chosen: - Interpretability over speed: Kept allocation 50/50 RCT rather than a bandit, to get clean ATEs and learn across segments; accepted a slightly slower convergence. - Latency budget over model complexity: Bounded reranking candidates and used lightweight features; deferred heavier context features to a follow-up. - Variance reduction (CUPED, stratification) over longer runtime: Invested upfront in design to hit MDE sooner without over-ramping. Why this maps to the JD skill: The project demonstrates end-to-end experimentation rigor—powering, randomization strategy, variance reduction, SRM detection, robust inference, guardrail governance—and translates results into product decisions with quantified impact and clear trade-offs.