Success Metrics and Holdout Design for Facebook's Original News Feed

You are one of Facebook's early product managers, and the company is launching its News Feed. How would you evaluate the launch? ```hint Ask what the feed replaces Think about how people kept up with friends before a feed existed, and which of those habits the feed is meant to change. Some numbers should rise, some should fall, and some must not move. ``` ### Clarifying Questions - Is the decision whether to launch the feed at all, or whether the launch that has happened is working? - Can some users or some school networks be held back from the feed for a while, or does it go to everyone at once? - Over what period should the launch be judged: the first days, or the months after people get used to it? - Is the main aim more interaction between friends, more people sharing, or more people coming back each day? ### What a Strong Answer Covers - A goal for the feed and a primary success metric tied to it, including behaviors the feed should grow and behaviors it should replace. - A measurement design that accounts for friends influencing each other, so the comparison is not contaminated. - Guardrails for privacy, user sentiment and sharing, and a way to tell an initial negative reaction from lasting harm. - How novelty effects are separated from durable changes in return visits and posting. - A decision rule agreed in advance that covers keeping, changing and rolling back the feed. ### Follow-up Questions - In the first week, protest groups form and complaints pour in, yet visits are up. What do you recommend? - Profile page views drop sharply after launch. Is that a problem? - How would you tell whether the feed makes people share more, rather than only read more?