Design experiments for payments, search, and promotions

You are a Data Scientist supporting a consumer marketplace app (users search restaurants and place orders). Answer the following product experimentation scenarios. For each scenario, explain: - The **goal / hypothesis** - **Primary metric** plus **diagnostic** and **guardrail** metrics (with tradeoffs) - **Experiment design**: treatment(s), control, randomization unit, eligibility, duration, sample-size/power considerations - Key **confounders / pitfalls** (e.g., selection bias, network effects, novelty, seasonality) and how you’d mitigate them - How you would interpret results and decide whether to launch ## Scenario A — Default payment method Today the app can automatically use **stored credit** to pay at checkout (vs making the user pick another method). The team wants to change how this “auto-use credit” behavior works (e.g., turn it on by default, change the UI, or change the logic). **Question:** How would you test the change? ## Scenario B — Search results duplicate listings In search results, sometimes the **#1 organic restaurant** is the same as the **sponsored restaurant**. Users then see **two identical listings** on the same results page. **Questions:** 1) Is this good or bad for users and the business? What would you measure to decide? 2) If you decide to change it (e.g., deduplicate, replace the sponsored slot, add labeling), how would you test the change? ## Scenario C — Promotions adoption and customization Restaurants can offer a promotion like **“$5 off $30”**. The business wants to: 1) **Increase restaurant adoption** of promotions. 2) Add **customizable promotions** (e.g., “$3 off $15”, “$10 off $50”) and design the feature + experiment. 3) Compare the new **customizable** option vs the existing fixed **“$5 off $30”** option: which is better? Assume this is a two-sided marketplace: restaurant behavior can affect user experience, and promotions may shift demand across restaurants.