Evaluate Model-Targeted Subscription Notifications

# Evaluate Model-Targeted Subscription Notifications A classification model identifies users likely to subscribe, and the product team plans to send those users a notification. Design a measurement and experimentation plan. Include checks before the experiment, how to interpret a flat result, and alternatives to a conventional user-level A/B test with their trade-offs. Explain how the plan separates model prediction quality from the incremental effect of sending notifications. ### What a Strong Answer Covers - Subscription and user-experience outcomes with a stable eligible-population denominator. - A design estimating notification incrementality rather than only predictive discrimination. - Instrumentation validation, A/A checks, allocation checks, and a prespecified analysis. - A diagnosis of flat effects and at least two alternative evaluation designs with assumptions. ```hint A likely subscriber may subscribe anyway High predicted conversion does not establish that the notification changes the user’s decision. ``` ### Follow-up Questions - How would you decide whether to target likely subscribers or users most persuadable by a notification? - What evidence can an offline classifier evaluation provide before any notification is sent?