Defining Success Metrics for Intercom's New SMS Channel

Quick Overview

A product analytics question about how to judge whether Intercom's new SMS engagement feature is succeeding. It tests defining goals for the businesses sending texts, their recipients and Intercom, choosing a primary metric with texting-specific guardrails, and separating causal impact from adoption bias.

Defining Success Metrics for Intercom's New SMS Channel

Company: Intercom

Role: Product Manager

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Other

Intercom has rolled out SMS as a new way to engage users through text messages. How would you decide whether this feature is a success? ```hint Success for whom More than one party has to get something out of a text before it counts as a win. Work out what each of them wants from SMS, then choose the measure that best reflects all of them at once. ``` ### Clarifying Questions - Is SMS a channel that businesses on Intercom use to text their own customers, or a way for Intercom to reach the people who use Intercom itself? - Does the feature cover two-way support conversations, outbound messages such as reminders and campaigns, or both? - Is SMS billed separately, for example per message, so that revenue and margin are part of success, or is it bundled into existing plans? - How long has the feature been live: are we judging early adoption or its long-term effect on the business? ### What a Strong Answer Covers - The goals SMS serves for the businesses sending messages, the people receiving them, and Intercom itself - One primary success metric with a precise definition and a reason it beats counting messages sent - Supporting metrics across setup, adoption, recipient engagement and business outcomes - Guardrails specific to texting: opt-outs, spam complaints, delivery failures, consent compliance and per-message cost - How to separate the feature's causal effect from the fact that healthier accounts adopt new features first ### Follow-up Questions - Reply rates on SMS are high, but opt-out rates keep climbing month over month. What do you conclude, and what would you change? - Most new SMS volume replaces emails that were already performing well. Is the feature still a success? - How would you show that adopting SMS improves customer retention rather than just correlating with it?

Overview: A product analytics question about how to judge whether Intercom's new SMS engagement feature is succeeding. It tests defining goals for the businesses sending texts, their recipients and Intercom, choosing a primary metric with texting-specific guardrails, and separating causal impact from adoption bias.

Intercom logo
Intercom
Dec 25, 2024
mediumProduct ManagerOtherAnalytics & Experimentation
0
0

Intercom has rolled out SMS as a new way to engage users through text messages. How would you decide whether this feature is a success?

Clarifying Questions Guidance

  • Is SMS a channel that businesses on Intercom use to text their own customers, or a way for Intercom to reach the people who use Intercom itself?
  • Does the feature cover two-way support conversations, outbound messages such as reminders and campaigns, or both?
  • Is SMS billed separately, for example per message, so that revenue and margin are part of success, or is it bundled into existing plans?
  • How long has the feature been live: are we judging early adoption or its long-term effect on the business?

What a Strong Answer Covers Guidance

  • The goals SMS serves for the businesses sending messages, the people receiving them, and Intercom itself
  • One primary success metric with a precise definition and a reason it beats counting messages sent
  • Supporting metrics across setup, adoption, recipient engagement and business outcomes
  • Guardrails specific to texting: opt-outs, spam complaints, delivery failures, consent compliance and per-message cost
  • How to separate the feature's causal effect from the fact that healthier accounts adopt new features first

Follow-up Questions Guidance

  • Reply rates on SMS are high, but opt-out rates keep climbing month over month. What do you conclude, and what would you change?
  • Most new SMS volume replaces emails that were already performing well. Is the feature still a success?
  • How would you show that adopting SMS improves customer retention rather than just correlating with it?