How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a easy difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Intuit.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Intuit during technical interviews.

Diagnose KPI anomaly and evaluate promotion/A-B test

Q: Diagnose KPI anomaly and evaluate promotion/A-B test

This question evaluates a data scientist's competencies in KPI anomaly detection, causal inference for promotion effectiveness, and A/B test design and analysis, including skills in data validation, segmentation, uplift estimation, bias mitigation, metric selection, and power calculations.

You are a Data Scientist supporting a TurboTax product team. You are asked to handle three related analytics tasks.

1) KPI anomaly investigation

A dashboard shows that yesterday:

start_to_file_conversion dropped from ~18% to ~12% (day-over-day).
Total traffic and marketing spend look roughly flat.

Your task

Describe a structured approach to:

Validate whether the drop is real vs. a data/definition/pipeline issue.
Localize the issue (which step, segment, platform, geo, channel, etc.).
Propose the most likely root causes and what data you’d pull to confirm.

2) Was last year’s promotion successful?

Last year a promotion offered $X off to some users. You have user-level historical data:

user_id , signup_date , device , channel
promo_exposed (whether user saw the promo)
redeemed (whether user redeemed)
Outcomes: started_return , filed_return , revenue , refund_amount
Pre-period behavior: prior_year_filed (0/1), prior_year_revenue

Your task

Explain how you would determine whether the promotion was “successful,” including:

Primary metric(s), diagnostic metrics, and guardrails.
How you’d estimate incremental lift (not just correlation), and what assumptions you need.
How you’d handle selection bias (e.g., promo shown more to likely filers).

3) A/B testing case

The PM wants to run an A/B test on a new onboarding flow intended to increase filing completion.

Your task

Design and analyze the experiment:

Unit of randomization and why.
Primary metric + guardrails.
Power/MDE approach (high-level is fine).
Analysis plan (e.g., handling outliers, multiple metrics, SRM, novelty/seasonality).
What you would recommend if the primary metric improves but a guardrail worsens.

You are a Data Scientist supporting a TurboTax product team. You are asked to handle three related analytics tasks.

1) KPI anomaly investigation

A dashboard shows that yesterday:

start_to_file_conversion dropped from ~18% to ~12% (day-over-day).
Total traffic and marketing spend look roughly flat.

Your task

Describe a structured approach to:

Validate whether the drop is real vs. a data/definition/pipeline issue.
Localize the issue (which step, segment, platform, geo, channel, etc.).
Propose the most likely root causes and what data you’d pull to confirm.

2) Was last year’s promotion successful?

Last year a promotion offered $X off to some users. You have user-level historical data:

user_id , signup_date , device , channel
promo_exposed (whether user saw the promo)
redeemed (whether user redeemed)
Outcomes: started_return , filed_return , revenue , refund_amount
Pre-period behavior: prior_year_filed (0/1), prior_year_revenue

Your task

Explain how you would determine whether the promotion was “successful,” including:

Primary metric(s), diagnostic metrics, and guardrails.
How you’d estimate incremental lift (not just correlation), and what assumptions you need.
How you’d handle selection bias (e.g., promo shown more to likely filers).

3) A/B testing case

The PM wants to run an A/B test on a new onboarding flow intended to increase filing completion.

Your task

Design and analyze the experiment:

Unit of randomization and why.
Primary metric + guardrails.
Power/MDE approach (high-level is fine).
Analysis plan (e.g., handling outliers, multiple metrics, SRM, novelty/seasonality).
What you would recommend if the primary metric improves but a guardrail worsens.

Diagnose KPI anomaly and evaluate promotion/A-B test

Quick Overview

1) KPI anomaly investigation

Your task

2) Was last year’s promotion successful?

Your task

3) A/B testing case

Your task

Solution

Comments (0)

Diagnose KPI anomaly and evaluate promotion/A-B test

Quick Overview

1) KPI anomaly investigation

Your task

2) Was last year’s promotion successful?

Your task

3) A/B testing case

Your task

Solution

Comments (0)