Landing Page Redesign A/B Test: Success Metric, Sample Size and Launch Rule

You are a product manager at Dropbox, and your team has built a new landing page. Design an experiment that shows whether the new page is a success. ```hint Begin with the page's job Decide which visitor action proves the page did its job before choosing anything else. Then work out who gets randomized, how many visitors you need, and which result would make you launch. ``` ### Clarifying Questions - Is this the main homepage that presents plans, or a campaign page that visitors from paid search or partners land on? - Does the new page replace an existing page for the same traffic, or is it a new destination? If it is new, where does that traffic go today? - What is the page meant to drive: free sign-ups, trial starts or paid purchases? - Roughly how much traffic does the page receive, and what share of visitors convert today? ### What a Strong Answer Covers - A hypothesis and a primary success metric tied to the page's purpose, with downstream quality and guardrail metrics - The unit of randomization and eligibility rules for mostly logged-out visitors - A sample-size and duration estimate based on baseline conversion and the smallest lift worth detecting - Validity checks during and after the test, and a decision rule agreed before launch - A plan for low traffic, or for metrics that point in different directions ### Follow-up Questions - Sign-ups go up, but fewer of the new sign-ups become paying customers. Do you launch? - The page gets only a few thousand visitors a week. How would you change the approach? - How would you run the test without hurting the page's ranking in search results?