Metric Framework for Instagram's Search Bar, From Query Intent to Satisfied Session

Quick Overview

A product analytics question about how to judge whether Instagram's search feature is succeeding. It tests separating user intents, building a metric funnel from search start to a satisfied outcome, choosing a north-star metric with guardrails, and handling clicks on results that do not match the query.

Metric Framework for Instagram's Search Bar, From Query Intent to Satisfied Session

Company: TikTok

Role: Product Manager

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Other

How would you tell whether search on Instagram is succeeding? ```hint Success depends on intent Separate the reasons people open search, such as finding a specific account versus browsing a topic, before you pick a single number. ``` ### Clarifying Questions - Does search here include every result type (accounts, keywords, hashtags, places and audio), or mainly finding accounts? - Is this an ongoing health assessment of search, or a readout for one change, such as a new ranking model? - Which outcome matters most to the business: new connections between people, discovery of creators and content, or overall time in the app? ### What a Strong Answer Covers - The role search plays in Instagram and the distinct user intents it serves - A metric funnel from starting a search to a satisfied outcome, with precise definitions and time windows - One north-star metric, why it was chosen, and how it could be gamed - Guardrails for speed, integrity of results and the health of the rest of the app - How to tell a click apart from real satisfaction, using post-click behavior, rater judgments or surveys ### Follow-up Questions - Suppose many clicks land on results that do not match the query. Would your metrics count those as success, and how would you correct for that? - Is it a problem if someone searches, gets results unrelated to their query and still engages with them? Does the answer depend on what they searched for? - After a ranking change, search success rate goes up but searches per user go down. How do you interpret that?

Overview: A product analytics question about how to judge whether Instagram's search feature is succeeding. It tests separating user intents, building a metric funnel from search start to a satisfied outcome, choosing a north-star metric with guardrails, and handling clicks on results that do not match the query.

TikTok logo
TikTok
Nov 14, 2024
mediumProduct ManagerOtherAnalytics & Experimentation
0
0

How would you tell whether search on Instagram is succeeding?

Clarifying Questions Guidance

  • Does search here include every result type (accounts, keywords, hashtags, places and audio), or mainly finding accounts?
  • Is this an ongoing health assessment of search, or a readout for one change, such as a new ranking model?
  • Which outcome matters most to the business: new connections between people, discovery of creators and content, or overall time in the app?

What a Strong Answer Covers Guidance

  • The role search plays in Instagram and the distinct user intents it serves
  • A metric funnel from starting a search to a satisfied outcome, with precise definitions and time windows
  • One north-star metric, why it was chosen, and how it could be gamed
  • Guardrails for speed, integrity of results and the health of the rest of the app
  • How to tell a click apart from real satisfaction, using post-click behavior, rater judgments or surveys

Follow-up Questions Guidance

  • Suppose many clicks land on results that do not match the query. Would your metrics count those as success, and how would you correct for that?
  • Is it a problem if someone searches, gets results unrelated to their query and still engages with them? Does the answer depend on what they searched for?
  • After a ranking change, search success rate goes up but searches per user go down. How do you interpret that?