Facebook Gaming Handheld: Setting Success Metrics and Deciding Launch When Sessions Shrink

Facebook is developing a handheld gaming device. Set the goals for the device and the metrics you would use to judge it. Then consider a follow-up: six months later, weekly active users (WAU) have risen by 10% while session length has fallen by 12%. Would you still ship it? ### Clarifying Questions - What does the device do beyond playing games: is playing and chatting with Facebook friends part of the concept, or is that ours to define? - How does the device make money: is it sold at a profit, or priced low with revenue expected from games and the platform? - Six months after what, a limited launch or a full market release? And does "ship" mean launching widely, continuing to sell, or releasing a change to the device? - What are the 10% and 12% measured against (a control group, the previous period, or a forecast), and whose WAU is it: device owners or Facebook gaming users in general? ### Part 1 — Set the goals and metrics Explain what the device should achieve for Facebook and for the people who buy it, choose a primary goal, and define the metrics you would track, including guardrails. ```hint Why would a social company make hardware? Start from what Facebook gains by owning a gaming device; that answer decides which kind of usage counts as success. ``` #### What This Part Should Cover - How the device ties to Facebook's mission and business, and one primary goal chosen over the alternatives. - A primary metric for that goal, with supporting metrics across the owner lifecycle. - Guardrails for the device experience, the business, and the effect on Facebook's other products. ### Part 2 — Decide whether to ship Six months in, WAU is up 10% and average session length is down 12%. Explain how you read these numbers, what you would check before deciding, and whether you would ship. ```hint Combine before you judge Work out what the two changes imply together for total time played, then ask whether existing players changed their behavior or a different mix of players arrived. ``` #### What This Part Should Cover - What the two changes imply together, and why shorter sessions could be good or bad on a handheld. - An ordered check of the data, the user segments and the likely causes. - A clear ship decision tied to the Part 1 goal, with the conditions that would change it. ### What a Strong Answer Covers - A Part 1 goal that is actually used to make the Part 2 decision. - Hardware-specific factors such as battery life, reliability, the game catalog and the cost of reversing a hardware launch. - Separation of shifts in the player mix from changes in how the same players behave. - A decision stated plainly, with the evidence that would reverse it. ### Follow-up Questions - You also learn that sessions per user fell over the same period. Does your decision change? - Almost all of the drop in session length comes from players who use the device with friends. How do you read that? - How would you tell whether the device grows people's total time with Facebook or only moves time away from the phone app?