A/B testing Facebook's swap of the posting box for an Instagram-style plus button

Quick Overview

An experimentation question about Facebook replacing its posting box with an Instagram-style plus button at the bottom of the screen. It tests defining success, picking posting and engagement metrics with guardrails, and designing an A/B test that handles novelty, user segments and network effects.

A/B testing Facebook's swap of the posting box for an Instagram-style plus button

Company: Meta

Role: Product Manager

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Other

Facebook is weighing a change to how people start a post. Today they tap into a posting box, the composer. The proposal borrows Instagram's pattern: remove the box and put a "+" button at the bottom of the screen instead. How would you test whether this is a good idea? ```hint Look past the first weeks Moving an entry point that people tap out of habit changes behavior in ways that can fade or grow over time. Think about who would notice the move most, and how long you would watch before trusting the result. ``` ### Clarifying Questions - What would make the change "good" for the team: more posts overall, more photo and video posts, more engagement on what gets posted, or a consistent design across Meta's apps? - Does the change apply only to the mobile apps or also to the desktop site, and only to the main feed or also to profiles and groups where a posting box appears? - Does the "+" open the same composer as today, or a different, media-first creation flow? - Are there limits on how many users the test can include or how long it can run? ### What a Strong Answer Covers - A definition of success and testable hypotheses about how the button changes posting and what friends then see - A primary content-creation metric, posting-funnel and downstream engagement metrics, and guardrails such as retention, time spent and ad revenue - An experiment design: randomization unit, interference between friends, stratified segments, sample size, and a duration that handles novelty and change aversion - How to read the results and decide: segment differences, effects that shift over time, launch criteria and a long-term holdout ### Follow-up Questions - Posts per user rise, but comments and reactions per post fall and time spent dips slightly. Do you launch? - Treatment users post more, and their friends in the control group see those extra posts. How does that bias your estimate, and how would you design around it? - The change helps people who also use Instagram and hurts people who do not. What do you do?

Overview: An experimentation question about Facebook replacing its posting box with an Instagram-style plus button at the bottom of the screen. It tests defining success, picking posting and engagement metrics with guardrails, and designing an A/B test that handles novelty, user segments and network effects.

Meta logo
Meta
Aug 26, 2024
mediumProduct ManagerOtherAnalytics & Experimentation
0
0

Facebook is weighing a change to how people start a post. Today they tap into a posting box, the composer. The proposal borrows Instagram's pattern: remove the box and put a "+" button at the bottom of the screen instead. How would you test whether this is a good idea?

Clarifying Questions Guidance

  • What would make the change "good" for the team: more posts overall, more photo and video posts, more engagement on what gets posted, or a consistent design across Meta's apps?
  • Does the change apply only to the mobile apps or also to the desktop site, and only to the main feed or also to profiles and groups where a posting box appears?
  • Does the "+" open the same composer as today, or a different, media-first creation flow?
  • Are there limits on how many users the test can include or how long it can run?

What a Strong Answer Covers Guidance

  • A definition of success and testable hypotheses about how the button changes posting and what friends then see
  • A primary content-creation metric, posting-funnel and downstream engagement metrics, and guardrails such as retention, time spent and ad revenue
  • An experiment design: randomization unit, interference between friends, stratified segments, sample size, and a duration that handles novelty and change aversion
  • How to read the results and decide: segment differences, effects that shift over time, launch criteria and a long-term holdout

Follow-up Questions Guidance

  • Posts per user rise, but comments and reactions per post fall and time spent dips slightly. Do you launch?
  • Treatment users post more, and their friends in the control group see those extra posts. How does that bias your estimate, and how would you design around it?
  • The change helps people who also use Instagram and hurts people who do not. What do you do?