# Diagnose KPI anomaly and evaluate promotion/A-B test You are a Data Scientist supporting a **TurboTax** product team. You are asked to handle three related analytics tasks. ## 1) KPI anomaly investigation A dashboard shows that yesterday: - `start_to_file_conversion` dropped from ~18% to ~12% (day-over-day). - Total traffic and marketing spend look roughly flat. ### Your task Describe a structured approach to: - Validate whether the drop is real vs. a data/definition/pipeline issue. - Localize the issue (which step, segment, platform, geo, channel, etc.). - Propose the most likely root causes and what data you’d pull to confirm. ## 2) Was last year’s promotion successful? Last year a promotion offered **$X off** to some users. You have user-level historical data: - `user_id`, `signup_date`, `device`, `channel` - `promo_exposed` (whether user saw the promo) - `redeemed` (whether user redeemed) - Outcomes: `started_return`, `filed_return`, `revenue`, `refund_amount` - Pre-period behavior: `prior_year_filed` (0/1), `prior_year_revenue` ### Your task Explain how you would determine whether the promotion was “successful,” including: - Primary metric(s), diagnostic metrics, and guardrails. - How you’d estimate **incremental lift** (not just correlation), and what assumptions you need. - How you’d handle selection bias (e.g., promo shown more to likely filers). ## 3) A/B testing case The PM wants to run an A/B test on a new onboarding flow intended to increase filing completion. ### Your task Design and analyze the experiment: - Unit of randomization and why. - Primary metric + guardrails. - Power/MDE approach (high-level is fine). - Analysis plan (e.g., handling outliers, multiple metrics, SRM, novelty/seasonality). - What you would recommend if the primary metric improves but a guardrail worsens. ### Constraints & Assumptions - Preserve the scope, facts, inputs, and requested outputs from the prompt above. - If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it. - Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate. ### Clarifying Questions to Ask - Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric. - State assumptions about instrumentation, randomization, sample size, and data quality. - Separate descriptive analysis from causal claims. ### What a Strong Answer Covers - A metric framework with primary, guardrail, and diagnostic metrics. - A credible analysis or experiment design with clear assumptions and bias checks. - SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant. - An actionable recommendation that explains trade-offs and next steps. ### Follow-up Questions - What sanity checks would you run before trusting the result? - How would you handle novelty effects, seasonality, or selection bias? - What decision would you make if metrics disagree?