Executive Readout for a Marketplace Recommendations A/B Test: Which Metrics to Show

You are the product manager for the recommendations feature of a large e-commerce marketplace, where many sellers list products and buyers browse and purchase them. Over the past quarter your team ran an A/B test of a new recommendations feature, and you are now due to walk the company's executives through the outcome. Which metrics would you show them, how would you organize them, and why would you pick those over others? ```hint Built for a decision The executives will use your readout to make a call. Work out which few numbers would settle that call, and which missing numbers would make them doubt it. ``` ### Clarifying Questions - What did the new feature change: where recommendations appear (home page, product pages, cart, emails), how they are ranked, or both? - What decision is the executive team being asked to make: roll out to every buyer, extend the test, or fund further work on recommendations? - Was a primary success metric agreed before the test began, and was traffic split by user or by session? - How does the marketplace make money: commission on each sale, seller-paid promoted listings, or both? ### What a Strong Answer Covers - One headline success metric tied to the marketplace's economics and chosen in advance, instead of a long undifferentiated list. - Recommendation-specific supporting metrics that explain why the headline moved, including whether sales were new or simply shifted from search. - Guardrails for buyers, sellers and long-term health, such as returns, seller concentration, retention and page speed. - Evidence the result is trustworthy: effect size with an interval, test validity checks, and how the lift behaved across the quarter. - A clear recommendation and a deck structure suited to executives. ### Follow-up Questions - Click-through on recommended items rose sharply, but revenue per buyer is flat. What do you tell the executives? - The lift is strong for new buyers but slightly negative for long-time buyers. Do you recommend launching? - An executive asks whether a full quarter was too long or too short for this test. How do you answer?