Design experiments for payments, search, and promotions

You are a Data Scientist supporting a consumer marketplace app (users search restaurants and place orders). Answer the following product experimentation scenarios. For each scenario, explain:

• The goal / hypothesis
• Primary metric plus diagnostic and guardrail metrics (with tradeoffs)
• Experiment design : treatment(s), control, randomization unit, eligibility, duration, sample-size/power considerations
• Key confounders / pitfalls (e.g., selection bias, network effects, novelty, seasonality) and how you’d mitigate them
• How you would interpret results and decide whether to launch

Scenario A — Default payment method

Today the app can automatically use stored credit to pay at checkout (vs making the user pick another method). The team wants to change how this “auto-use credit” behavior works (e.g., turn it on by default, change the UI, or change the logic).

Question: How would you test the change?

Scenario B — Search results duplicate listings

In search results, sometimes the #1 organic restaurant is the same as the sponsored restaurant. Users then see two identical listings on the same results page.

Questions:

1. Is this good or bad for users and the business? What would you measure to decide?
2. If you decide to change it (e.g., deduplicate, replace the sponsored slot, add labeling), how would you test the change?

Scenario C — Promotions adoption and customization

Restaurants can offer a promotion like “$5 off$30”. The business wants to:

1. Increase restaurant adoption of promotions.
2. Add customizable promotions (e.g., “ $3 off$ 15”, “ $10 off$ 50”) and design the feature + experiment.
3. Compare the new customizable option vs the existing fixed “$5 off$30” option: which is better?

Assume this is a two-sided marketplace: restaurant behavior can affect user experience, and promotions may shift demand across restaurants.