How to experiment on ETA reduction

Q: How to experiment on ETA reduction

This question evaluates a data scientist's competence in causal inference, A/B test design, metric selection, and diagnosing observational confounding when reducing ETA might affect conversion.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

Loading...

Context

You work on a consumer app where users see an ETA (estimated time to arrival/delivery) during a funnel (e.g., browsing → checkout → order placed). The team can change product and/or operations to reduce ETA.

You have historical data showing a positive correlation between ETA and conversion (users are more likely to convert when ETA is higher), which seems counterintuitive.

Questions

Why might reducing ETA be beneficial? List plausible business/product benefits and what metrics they would affect.
Why might ETA be positively correlated with conversion in observational data? Provide multiple hypotheses (confounding/selection effects) and how you would test or rule them out.
Design an experiment to estimate the causal impact of reducing ETA on conversion.
- Define the treatment , control , and randomization unit .
- Specify primary metric , diagnostic metrics , and guardrail metrics .
- Discuss risks like interference, seasonality, and delayed effects.
Confidence intervals: Explain how you would compute and interpret a confidence interval for the conversion lift.
Unit of analysis: Why might the analysis not be done at the user level? When is user-level appropriate vs session/order/request-level?

How to experiment on ETA reduction

Quick Overview

Context

Questions

Solution

Comments (0)