Uber is considering a redesign of the pickup ETA card shown to riders after they request a trip. The hypothesis is that clearer ETA presentation reduces cancellations and increases completed trips. Design the experiment end-to-end. Provide: - Experimental unit and randomization: Choose rider-level, request-level, or geo/time cluster; justify and discuss interference risks (e.g., supply-side coupling, surge, driver behavior) and how you would mitigate them. - Target population and eligibility: Precisely define who is included/excluded (e.g., first-time vs repeat riders, specific cities, iOS/Android versions), and how to handle riders with multiple requests in the window. - Primary metric and guardrails: Pick a single primary metric; propose 2–3 guardrails (e.g., driver acceptance rate, average wait time, surge incidence). Define each precisely, including numerator/denominator and window. - Instrumentation plan: List exact events/fields to log (e.g., request_id, rider_id, device_id, city_id, treatment_arm, event_name, timestamp_ms, cancel_reason, trip_started, trip_completed, driver_supply_at_request, ETA_shown). Include sampling rate, idempotency, and assignment audit (SRM checks). - Sample size and duration: Assume baseline booking-to-trip-completion rate p0 = 0.120, minimum detectable relative lift = +5%, two-sided α = 0.05, power = 0.80, 1:1 split, and 200,000 eligible requests per day (stable). Compute the per-arm sample size for a difference-in-proportions test and the expected experiment duration (days) to reach it, accounting for data loss of 3%. - Ramp strategy: Propose a safe ramp (e.g., 1%→10%→50%→100%), pre-specified stopping rules, and monitoring (including SRM and guardrails) at each ramp. - Bias/variance controls: Describe how you would handle seasonality, city heterogeneity (e.g., stratification or CUPED with pre-period completion), and bot/abuse filtering. - Success criteria and rollout: Pre-register decision thresholds and what to do if primary improves but a guardrail degrades. State any additional assumptions you need and show the sample size math clearly.

This question evaluates experimental design and causal inference skills, covering A/B testing, randomization and interference considerations, metric selection and guardrails, instrumentation and logging, sample size calculation, ramping and bias/variance controls in the Analytics & Experimentation domain.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Uber.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Uber during technical interviews.

Design an Uber A/B experiment end-to-end

Experiment Design: Pickup ETA Card Redesign

Context: After a rider requests a trip, the app shows a pickup ETA card. The hypothesis is that clearer ETA presentation reduces cancellations and increases completed trips. Design an end-to-end A/B test to evaluate this.

Provide the following:

Experimental unit and randomization

Choose one: rider-level, request-level, or geo/time cluster.
Justify your choice and discuss interference risks (e.g., supply-side coupling, surge dynamics, driver behavior) and how you would mitigate them.

Target population and eligibility

Precisely define inclusion/exclusion (e.g., first-time vs repeat riders, specific cities, iOS/Android versions).
State how you will handle riders with multiple requests during the experiment window.

Primary metric and guardrails

Select a single primary metric.
Propose 2–3 guardrails (e.g., driver acceptance rate, average wait time, surge incidence).
Define each metric precisely: numerator, denominator, and measurement window.

Instrumentation plan

List exact events/fields to log, e.g., request_id, rider_id, device_id, city_id, treatment_arm, event_name, timestamp_ms, cancel_reason, trip_started, trip_completed, driver_supply_at_request, ETA_shown.
Include sampling rate, idempotency strategy, and assignment audit plan (including SRM checks).

Sample size and duration

Assume: baseline booking-to-trip-completion rate p0 = 0.120; minimum detectable relative lift = +5%; two-sided α = 0.05; power = 0.80; 1:1 split; 200,000 eligible requests per day (stable).
Compute the per-arm sample size for a two-sample difference-in-proportions test and the expected experiment duration (days) to reach it, accounting for 3% data loss.
Show the sample size math clearly.

Ramp strategy

Propose a safe ramp (e.g., 1% → 10% → 50% → 100%), pre-specified stopping rules, and monitoring (including SRM and guardrails) at each ramp.

Bias/variance controls

Describe controls for seasonality, city heterogeneity (e.g., stratification or CUPED with pre-period completion), and bot/abuse filtering.

Success criteria and rollout

Pre-register decision thresholds and actions if the primary metric improves but a guardrail degrades.

State any additional assumptions you need.

Experiment Design: Pickup ETA Card Redesign

Provide the following:

Experimental unit and randomization

Choose one: rider-level, request-level, or geo/time cluster.
Justify your choice and discuss interference risks (e.g., supply-side coupling, surge dynamics, driver behavior) and how you would mitigate them.

Target population and eligibility

Precisely define inclusion/exclusion (e.g., first-time vs repeat riders, specific cities, iOS/Android versions).
State how you will handle riders with multiple requests during the experiment window.

Primary metric and guardrails

Select a single primary metric.
Propose 2–3 guardrails (e.g., driver acceptance rate, average wait time, surge incidence).
Define each metric precisely: numerator, denominator, and measurement window.

Instrumentation plan

List exact events/fields to log, e.g., request_id, rider_id, device_id, city_id, treatment_arm, event_name, timestamp_ms, cancel_reason, trip_started, trip_completed, driver_supply_at_request, ETA_shown.
Include sampling rate, idempotency strategy, and assignment audit plan (including SRM checks).

Sample size and duration

Assume: baseline booking-to-trip-completion rate p0 = 0.120; minimum detectable relative lift = +5%; two-sided α = 0.05; power = 0.80; 1:1 split; 200,000 eligible requests per day (stable).
Compute the per-arm sample size for a two-sample difference-in-proportions test and the expected experiment duration (days) to reach it, accounting for 3% data loss.
Show the sample size math clearly.

Ramp strategy

Propose a safe ramp (e.g., 1% → 10% → 50% → 100%), pre-specified stopping rules, and monitoring (including SRM and guardrails) at each ramp.

Bias/variance controls

Describe controls for seasonality, city heterogeneity (e.g., stratification or CUPED with pre-period completion), and bot/abuse filtering.

Success criteria and rollout

Pre-register decision thresholds and actions if the primary metric improves but a guardrail degrades.

State any additional assumptions you need.

Design an Uber A/B experiment end-to-end

Quick Overview

Experiment Design: Pickup ETA Card Redesign

Solution

Comments (0)

Design an Uber A/B experiment end-to-end

Quick Overview

Experiment Design: Pickup ETA Card Redesign

Solution

Comments (0)