North Star, Quality and Guardrail Metrics for AI Assistance in Meta's Chat Apps

You are a PM at Meta responsible for the AI-powered features in its messaging apps. How would you define success for these features, and which metrics would you track? ```hint Separate activity from help A person who has to ask the AI three times sends more messages than one who got what they needed on the first try. Look for signals inside the conversation that show the AI finished the job the user had in mind. ``` ### Clarifying Questions - Which features are in scope: the Meta AI assistant that users chat with directly, AI help inside chats with friends (suggested replies, rewriting, summaries, generated images), or AI agents that answer customers on behalf of businesses? - Which apps are we measuring across: WhatsApp, Messenger, Instagram Direct, or all three, and does end-to-end encryption limit which conversation signals we can log? - Are these features newly launched, so adoption is the open question, or established, so depth and quality matter more? - Is there a business objective attached, such as more business messaging or deeper engagement, or is user value the only goal for now? ### What a Strong Answer Covers - The user value AI adds to messaging and how it connects to Meta's goals for its messaging apps - A north-star metric that counts successful AI-assisted outcomes rather than opens, prompts or AI messages sent - A funnel of supporting metrics (reach, trial, repeat use, depth), broken out by feature, app and segment - Quality, safety and trust measures specific to generative AI, collected in a way that respects private conversations - Guardrails that protect person-to-person messaging, the core product the AI sits inside ### Follow-up Questions - People use AI reply suggestions heavily, but messages between friends per user dip slightly in the treatment group. How do you decide whether the feature is good for Meta? - How would you measure answer quality at scale when you cannot read users' private chats? - The north star climbs after launch, but nearly all of the gain comes from AI image generation. What would you check before calling the launch a success?