Facebook is considering launching a new feature that allows users to schedule a post to be published at a future time. The product hypothesis is that scheduled posts will increase meaningful engagement by helping users publish content at better times and plan posts in advance.
You are asked to evaluate the feature before launch. In particular, the interviewer asks about both statistics and product measurement:
-
How would you define and estimate the failure rate of scheduled posts using a Bayesian approach?
-
How would you decide whether the feature is successful from a product perspective?
-
What experiment would you run, and what metrics would you use?
-
What biases, confounders, and edge cases would you watch for?
Assume a scheduled post is considered a technical failure if the user schedules the post and does not cancel it, but the system fails to publish it within 5 minutes of the scheduled time.