How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Upstart.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Upstart during technical interviews.

Decide to ship a signup experiment | Upstart Interview Question

Quick Overview

This question evaluates A/B testing and experimentation competencies, including data quality validation, metric selection, statistical estimation, power analysis, temporal and cohort effects, and decision-making trade-offs, and it falls under the Analytics & Experimentation domain for Data Scientist roles.

Context and Data

You are analyzing an A/B experiment for a redesigned user signup flow. The dataset includes the following columns per user/session or per aggregation unit: variant (A/B), sessions, signups, activation_d7, p95_latency_ms, support_tickets, refund_rate, revenue_d30.

Assumptions to make explicit and verify:

Randomization unit: user_id (sticky assignment). If only session-level logs exist, cluster by user_id.
Definitions:
- sessions: count of sessions exposed to the variant.
- signups: count of completed signups.
- activation_d7: count of users who activated within 7 days of signup (define “activation” precisely for your product).
- p95_latency_ms: 95th percentile request latency during signup flow (computed from request-level logs).
- support_tickets: count of tickets attributable to the signup/onboarding experience.
- refund_rate: refunds per activated or paying user (clarify denominator; use consistent unit across variants).
- revenue_d30: total revenue within 30 days from users exposed (define whether revenue is per user, per activated user, or all exposed; prefer per-user for inference).

Goal: Decide whether to ship the redesign using a principled testing plan with data quality checks, metric selection, estimation, power, temporal effects, and decision criteria.

Tasks

Describe exactly how you would:

Validate data quality (SRM test, bucketing integrity, exposure logs vs analytics counts).
Choose primary and guardrail metrics and justify them.
Compute effects with confidence intervals (including ratio metrics and non-parametric options if skewed), and apply variance reduction (e.g., CUPED) if appropriate.
Check power/min detectable effect and whether the observed duration met the pre-registered stopping rule.
Evaluate novelty and learning effects (time-sliced and cohort views).
Make the ship/no-ship call with a concrete decision framework that balances activation gains vs increased latency/support tickets.
List at least three additional insights to extract beyond the ship decision (e.g., segment heterogeneity by traffic source/device, step-drop analysis, form-field sensitivity).

Quick Overview

Context and Data

Assumptions to make explicit and verify:

Randomization unit: user_id (sticky assignment). If only session-level logs exist, cluster by user_id.

Definitions:

sessions: count of sessions exposed to the variant.
signups: count of completed signups.
activation_d7: count of users who activated within 7 days of signup (define “activation” precisely for your product).
p95_latency_ms: 95th percentile request latency during signup flow (computed from request-level logs).
support_tickets: count of tickets attributable to the signup/onboarding experience.
refund_rate: refunds per activated or paying user (clarify denominator; use consistent unit across variants).
revenue_d30: total revenue within 30 days from users exposed (define whether revenue is per user, per activated user, or all exposed; prefer per-user for inference).

Goal: Decide whether to ship the redesign using a principled testing plan with data quality checks, metric selection, estimation, power, temporal effects, and decision criteria.

Tasks

Describe exactly how you would:

Validate data quality (SRM test, bucketing integrity, exposure logs vs analytics counts).

Choose primary and guardrail metrics and justify them.

Compute effects with confidence intervals (including ratio metrics and non-parametric options if skewed), and apply variance reduction (e.g., CUPED) if appropriate.

Check power/min detectable effect and whether the observed duration met the pre-registered stopping rule.

Evaluate novelty and learning effects (time-sliced and cohort views).

Make the ship/no-ship call with a concrete decision framework that balances activation gains vs increased latency/support tickets.

List at least three additional insights to extract beyond the ship decision (e.g., segment heterogeneity by traffic source/device, step-drop analysis, form-field sensitivity).

Decide to ship a signup experiment

Quick Overview

Context and Data

Tasks

Solution

Comments (0)

Decide to ship a signup experiment

Quick Overview

Context and Data

Tasks

Solution

Comments (0)

Decide to ship a signup experiment

Quick Overview