Diagnose KPI anomaly and evaluate promotion/A-B test

You are a Data Scientist supporting a TurboTax product team. You are asked to handle three related analytics tasks.

1) KPI anomaly investigation

A dashboard shows that yesterday:

• start_to_file_conversion dropped from ~18% to ~12% (day-over-day).
• Total traffic and marketing spend look roughly flat.

Your task

Describe a structured approach to:

• Validate whether the drop is real vs. a data/definition/pipeline issue.
• Localize the issue (which step, segment, platform, geo, channel, etc.).
• Propose the most likely root causes and what data you’d pull to confirm.

2) Was last year’s promotion successful?

Last year a promotion offered $X off to some users. You have user-level historical data:

• user_id , signup_date , device , channel
• promo_exposed (whether user saw the promo)
• redeemed (whether user redeemed)
• Outcomes: started_return , filed_return , revenue , refund_amount
• Pre-period behavior: prior_year_filed (0/1), prior_year_revenue

Your task

Explain how you would determine whether the promotion was “successful,” including:

• Primary metric(s), diagnostic metrics, and guardrails.
• How you’d estimate incremental lift (not just correlation), and what assumptions you need.
• How you’d handle selection bias (e.g., promo shown more to likely filers).

3) A/B testing case

The PM wants to run an A/B test on a new onboarding flow intended to increase filing completion.

Your task

Design and analyze the experiment:

• Unit of randomization and why.
• Primary metric + guardrails.
• Power/MDE approach (high-level is fine).
• Analysis plan (e.g., handling outliers, multiple metrics, SRM, novelty/seasonality).
• What you would recommend if the primary metric improves but a guardrail worsens.