Design experiments for marketplace product changes

You are interviewing for a Data Scientist role at a food-delivery marketplace such as DoorDash. For each scenario below, explain how you would evaluate the change from an experimentation and product-analytics perspective.

For every scenario, discuss:

• the core hypothesis,
• the treatment and control definitions,
• the unit of randomization,
• the primary success metric,
• important guardrail metrics,
• likely sources of bias, interference, or spillover,
• how you would interpret results and make a launch decision.

Assume the marketplace has three sides: customers, restaurants, and advertisers/sponsored listings. Unless you justify otherwise, assume metrics are measured over a 14-day post-exposure window and that experiments use sticky assignment.

Scenario A: Auto-apply customer credits at checkout Customers can use available credits when placing an order. Today they must manually choose to apply credits. The product team wants to change the checkout flow so that available credits are applied automatically by default.

• How would you test this change?
• What should the primary metric be?
• Should you randomize by user, session, or order?
• What tradeoffs might appear between checkout conversion, customer experience, credit burn, platform margin, and future retention?

Scenario B: Duplicate restaurant in search results In search, the top organic restaurant and the sponsored restaurant can sometimes be the same merchant, so the user sees two nearly identical listings on the same page.

• Is this duplication good or bad for the marketplace?
• If you want to remove the duplicate, how would you test the change?
• What metrics would you use for user experience, ad performance, and overall marketplace value?
• How would you reason about the fact that removing the duplicate changes exposure for lower-ranked restaurants?

Scenario C: Restaurant promotions Restaurants can currently create a fixed promotion such as 5 dollars off 30 dollars. The company is considering three related questions:

1. How would you increase restaurant adoption of promotions?
2. If restaurants are allowed to customize the promotion format, for example 3 dollars off 15 dollars or 10 dollars off 50 dollars, how would you design and test that product?
3. If you want to compare the new customizable-promotion product against the old fixed-template product, what experiment or analysis would you run?

In Scenario C, consider both restaurant-side outcomes and customer-side outcomes. Discuss heterogeneity by restaurant size, cuisine, order volume, and new versus existing merchants. Also explain how you would separate incremental lift from cannibalization and adverse selection.