Reading a Feature Rollout Report: Mix Shifts, Low-Rate Platforms and Small Samples

A new feature is rolling out on several platforms. A weekly report shows, for each platform, the feature's creation rate, the number of devices behind that rate, and the figures for both the current week and the previous week. A product manager raises three questions about the latest report. Assume the creation rate is the share of devices in the rollout that created at least one item with the feature during the week; confirm the exact definition before answering. ### Clarifying Questions - Is the rate computed over every device enrolled so far, or only over devices that were active during the week? - Are devices added in waves, and were the newly added devices selected differently from earlier ones (region, device model, type of user)? - How long has each device had the feature when it is counted? - How many devices are behind each platform's rate? ### Part 1 — A falling rate on a growing platform On one platform, the creation rate is lower than the previous week, while the number of devices in the rollout has grown a lot. The PM thinks this means there is a retention problem. Is that conclusion right? ```hint Who is in the denominator this week? Think about how long the newly added devices have had the feature, and who they are, compared with the devices that were already there. ``` #### What This Part Should Cover - Why a pooled weekly rate can fall without any change in how existing devices behave - The cuts and checks that separate a mix change from a real decline - What metric would actually show a retention problem ### Part 2 — Two low-rate platforms Two other platforms have relatively low rates, and the PM wants to prioritize improving them. How would you evaluate that plan? ```hint Rate versus opportunity A low rate tells you where the feature performs worst, not where an improvement would add the most. ``` #### What This Part Should Cover - Whether the gap is real and the platforms are compared fairly - Diagnosing why the rate is low - Prioritizing by expected absolute impact and feasibility rather than by rate alone ### Part 3 — A high rate on a small sample One platform has the highest rate, but its sample is small. If the team wants to prioritize resources for that platform because of its rate, how do you view that statistically? ```hint The best of many When several platforms are compared, think about which one is most likely to land on top by chance. ``` #### What This Part Should Cover - The uncertainty of a rate estimated from few devices - Selection of the highest observed rate and regression to the mean - What evidence and what kind of estimate should drive the resource decision ### What a Strong Answer Covers - Separates changes in composition from changes in behavior, with a cohort view of the data - Quantifies uncertainty with intervals and recognizes when a sample is too small to rank on - Ties prioritization to absolute impact, cause and feasibility rather than to the ordering of rates - Gives the PM a clear recommendation and a better way to read the report ### Follow-up Questions - How would you redesign the weekly report so that these misreadings become less likely? - The PM wants a single "is the rollout healthy" number per platform. What would you provide? - With several platforms each week, one of them will always look worst. How do you avoid chasing noise from week to week?