Learn complex topic fast under deadline

# Example Context Scope: A high-stakes product review needed a causal estimate of a policy change on DAU and 7‑day retention. A randomised experiment had been aborted due to an infrastructure issue, so I needed a credible observational method within a week. New framework: Bayesian Structural Time Series (BSTS) via the CausalImpact approach to construct a counterfactual from control series and estimate treatment effect with uncertainty. Stakeholders: Product, Eng, Data Science, and an executive sponsor expecting a go/no-go decision for full rollout. Assumptions (made explicit up front): - Pre-period is long enough to learn seasonality/trends (≥90 days). - Covariates are predictive and not affected by treatment (no leakage). - The post-period is short (≈2 weeks), so uncertainty will be non-trivial. # Plan by Day (5 working days) Day 1 — Problem framing and feasibility - Clarify decision question, KPIs, and acceptable uncertainty (e.g., “OK if 95% interval width ≤2pp”). - Inventory data: outcome series, candidate control series (peer geos, older cohorts), known events (holidays, outages). - Skim core materials: CausalImpact vignettes, BSTS model structure (local level/trend, seasonality, spike-and-slab regression), leakage pitfalls. - Define MVE (minimum viable estimate): one primary KPI (DAU), one secondary (7‑day retention), a single model variant, and at least two high-ROI validation checks. Day 2 — First working prototype - Build a reproducible notebook using a well-tested library (R CausalImpact or Python causal_impact equivalent). - Preprocess: align calendars, remove outlier days, z-score covariates, hold back a validation window. - Fit baseline model with 120-day pre-period and 14-day post-period; include weekday seasonality and 5–10 control series. - Produce first cut: point estimate, 95% posterior intervals, and time-series plots. Day 3 — Validation and leakage control - Placebo tests: rotate “treated” label across comparable geos or time windows; ensure the observed effect is in the extreme tail of placebo distribution. - Backtesting: rolling-origin forecasts entirely within pre-period; compute RMSPE and calibration of posterior intervals. - Sensitivity: remove top predictors; vary pre-period length; check prior sensitivity for state components. - Quick triangulation with a simpler method (two-way fixed-effects DiD with geo and calendar FE) to see if effect direction and magnitude roughly agree. Day 4 — Harden and communicate - Lock covariate set after leakage checks (exclude any series that moved post-treatment). - Document assumptions, diagnostics (RMSPE, coverage), and sensitivity grids. - Create decision-ready artifacts: 1-page exec summary, appendix slides with diagnostics, and a reproducible notebook. - Peer review: 30-minute DS peer pass for sanity and failure-mode audit. Day 5 — Final review and contingency - Dry run with PM/Eng to align on interpretation and caveats. - Precompute alternative slices (by platform or region) only if they meet minimum sample thresholds. - Prepare contingency slide with trade-offs if asked to expand/contract scope. # Minimum Viable Artifacts (produced) - 1-page decision memo: question, method, headline effect, uncertainty, assumptions, and recommended decision. - Reproducible notebook: data prep, model fit, diagnostics, and plots (seeded for determinism). - Data dictionary: sources, transformations, and filters. - Validation checklist: placebo/backtest results, sensitivity table, leakage checks. - 5–7 slide deck for execs; appendix with methodology. # Validation Under Time Pressure - Pre-period fit quality: RMSPE and posterior predictive checks. Target RMSPE <3% for DAU; <5% for retention. - Placebo tests: Effect percentile vs. 50+ placebo geos/windows; empirical p-value. - Backtests: Rolling-origin 7-day forecasts; coverage of 95% intervals >90%. - Triangulation: Compare with DiD (two-way FE). Directional agreement and overlapping intervals are a positive signal. - Leakage checks: Remove any covariate with significant post-period shift correlated with treatment. Illustrative results (numbers from the actual run): - Pre-period RMSPE (DAU): 2.1%; coverage in backtests: 92%. - Estimated DAU lift: +1.8% [0.5%, 3.0%] over 14 days; retention: +0.4pp [0.1pp, 0.7pp]. - Placebo distribution: observed effect at 97th percentile (empirical p≈0.03). - DiD estimate: +1.6% [0.2%, 3.1%] — intervals overlap and direction matches. # Trade-offs Made - Chose a well-documented off-the-shelf BSTS over a custom model to save time; accepted limited hyperparameter exploration. - Restricted to one primary KPI and one secondary to keep validation focused. - Curated 8 high-signal control series to reduce computation and overfitting risk; did not pursue automated feature search beyond spike-and-slab. - Short post-period (14 days) increased uncertainty; accepted wider intervals with stronger placebo evidence. # Objective Signals of Success/Failure Success signals observed: - Quantitative: low RMSPE, good coverage, strong placebo separation, triangulation agreement. - Process: peer review passed with only minor comments; reproducibility confirmed by reruns (same seed → same summary stats). - Outcome: Executive accepted recommendation; subsequent staged geo A/B (two weeks later) yielded +1.5% [0.4%, 2.6%], within our credible interval. Potential failure signals to watch: - High pre-period error or poor coverage; effect not distinguishable from placebo distribution; large swings under small modeling changes; covariate leakage detected. # How I’d Update If Deadline Moved Up by 48 Hours Time compression strategy (focus on decision usefulness, not exhaustiveness): - Scope cut: Only DAU, single geo-grouping, and a single model variant. - Method simplification: Start with two-way FE DiD (geo and calendar FE, robust SEs). Use BSTS only if DiD diagnostics pass quickly and time permits. - Validation triage: Keep the two highest-ROI checks — placebo over time windows and pre-period backtest; drop extended sensitivity grid. - Covariate discipline: Use a pre-vetted set of controls (historically predictive, known non-reactive) to avoid leakage work. - Communication: Deliver a 1-page decision with clear caveats and a risk register; propose a follow-up validation plan post-decision. Compressed 3-day plan: - Day 1: Frame question, assemble data, run DiD baseline with diagnostics; produce early read. - Day 2: Run minimal BSTS with vetted controls; quick placebo and backtest; reconcile with DiD. - Day 3: Finalize memo and slides; peer spot-check; deliver. Guardrails in the compressed plan: - Precommit to thresholds (e.g., require placebo p<0.1 and backtest coverage >85%); if not met, escalate uncertainty and recommend staged rollout rather than full launch. # Pitfalls and How I Avoided Them - Covariate leakage: excluded any series showing contemporaneous jumps post-policy; preference for upstream, non-treated signals. - Seasonality and holidays: included weekday factors and explicit holiday dummies. - Data quality: monitored missingness/outliers; winsorized extreme days with documented rationale. - Over-interpretation: reported point estimates with credible intervals and emphasized decision thresholds, not single-number precision.