Define and Measure Merchant Variety's Impact on Consumers
Overview
This question evaluates competencies in marketplace analytics, experimental design, metric definition, and causal inference by requiring a measurable definition of merchant variety, specification of merchant- and consumer-level success metrics and guardrails, and an A/B test design; it is categorized under Analytics & Experimentation and demands both conceptual understanding and practical application. It is commonly asked to determine whether a candidate can translate product concepts into measurable KPIs, anticipate interference and operational constraints in experiments, and demonstrate rigor in powering and analyzing randomized tests that inform marketplace decisions.
Scenario
You are working in a two‑sided on‑demand delivery marketplace where "merchants" are restaurants/retailers and consumers browse a catalog to place orders. The product team is exploring how merchant variety (also called selection) affects consumer experience and marketplace health.
Question
- Define merchant variety/selection in this marketplace. Provide an actionable, measurable definition.
- List success metrics you would track at both the merchant level and the consumer level. Call out any marketplace/operational guardrails.
- Describe how you would design an A/B test to evaluate a change intended to increase variety. Address unit of randomization, interference risks, metrics, power, and analysis.
Hints: Consider category breadth, inventory depth, conversion, retention, order frequency, revenue per user.
Solution
Login required