Experiment Design for a Personalized Offer Recommendation System in a Rewards App

A consumer rewards app, where users earn rewards and can redeem offers shown in the app, plans to launch a personalized offer recommendation system that chooses which offers each user sees and in what order. Design the experiment for this launch: choose the metrics, design the test, and explain what you would decide under different outcomes. The report describes this round as product sense plus decisions under different situations but does not list the situations; the ones in Part 3 are examples to reason through. ### Clarifying Questions - What does each user see today: the same offers for everyone, offers ranked by simple rules, or something else? - Who pays for the reward attached to an offer: the app itself, or a partner that wants the offer redeemed? - Do offers have limited budgets or caps on the number of redemptions? - What is the main goal of the launch: more engagement and retention, more redemptions, or more revenue per user? ### Part 1 — Metrics Which metric would decide the launch, and which secondary metrics and guardrails would you track? ```hint Two sides of an offer An offer that users love can still cost more than it earns. Consider everyone affected when an offer is redeemed. ``` #### What This Part Should Cover - A primary metric tied to the launch goal, with a justification - Secondary metrics along the offer funnel and for user engagement - Guardrails on reward cost, offer economics and user experience ### Part 2 — Experiment design Design the A/B test: the unit of randomization, the control experience, the allocation, the sample size and duration, and the threats to validity. ```hint Shared budgets If an offer's budget runs out during the test, ask which group used it up and what the other group sees afterward. ``` #### What This Part Should Cover - The randomization unit and what control users see - Power, duration, and novelty or learning effects of a personalization model - Interference through shared offer budgets or training data, and how to handle it ### Part 3 — Decisions under different outcomes What would you recommend in each of these situations, and what analysis would you do before deciding? 1. Redemptions per user rise, but the total cost of rewards paid out rises faster than revenue. 2. The overall effect on the primary metric is flat, but new users improve and long-tenured users decline. 3. The primary metric improves strongly in the first week, and the gain shrinks by the fourth week. ```hint Look below the average For each situation, decide which breakdown or additional metric would tell you whether the effect is real, profitable and lasting. ``` #### What This Part Should Cover - A decision for each situation that follows from the stated goal and guardrails - The diagnostic analysis behind each decision - Options short of a full launch, such as a targeted launch, another iteration or a long-term holdout ### What a Strong Answer Covers - Metrics tied to the economics of offers, not only to clicks or redemptions - A valid design that accounts for shared offer budgets and for how the model learns - Decision rules fixed before the test starts - Segment and time-trend analysis that explains uneven or fading effects - Long-term measurement after launch ### Follow-up Questions - The recommendation model retrains daily on all users' interactions, including control users' behavior. Does that compromise the experiment? - How would you evaluate a new version of the model offline before spending traffic on another A/B test? - Under personalization, some offers receive almost no impressions. Why does that matter, and how would you measure and address it?