Meta Senior Data Scientist Interview Experience — Notification Product Sense Plus Two SQL Questions, Rejected

1. Product Sense — Notification Quality

Background: Cycle is a travel content sharing app. It has Content Recommendation notifications, Social notifications and Safety notifications.

Q1. How do you measure Notification Quality?
My answer: I first confirmed whether the notification system had already launched, and the interviewer said it had. I proposed User Engagement and Notification CTR.

Q2. Why use a 7-day CTR?
My answer: I had originally proposed a 7-day window. The interviewer asked back whether a click after 7 days is still reasonable. I changed my answer: first analyze the click delay distribution, then decide the attribution window. [Haha, I think I'd practiced too much SQL and just blurted out "7 days" without thinking...]

Q3. Besides CTR, what other metrics?
My answer: Hide Rate to measure negative user feedback. Failure Rate to measure technical failures or slow responses after a click. The interviewer pointed out it should be called Error Rate, and I accepted the correction.

Q4. If there had been no notification system before, how would you measure the incremental value?
My answer: I proposed a before/after comparison of overall user engagement, using basic statistics, confidence intervals and statistical significance. Then I went further and proposed A/B testing.

Q5. How would you design the A/B test? Which metrics?
My answer: User-level randomization, splitting users into Control and Treatment. I proposed notification CTR and notification Hide Rate. The interviewer kept pushing on how to compute CTR for Control when Control gets no notifications, and on what other metrics there could be. I kept explaining user-level randomization and did not come up with any other comparable primary metric.

Q6. What if CTR and Hide Rate both go up?
My answer: First check statistical significance and confidence intervals, don't roll out right away, then break down by user segment to find out why Hide Rate went up.

2. SQL Q1 — Survey Response Rate

Schema:
survey(userid, date, survey_event, response)
survey_event: 'impression' or 'response'
response: a rating from 1 to 5

Q1. Calculate the survey response rate.
My answer: I first calculated it at the impression level. The interviewer asked how to calculate it at the unique user level. I used COUNT(DISTINCT userid), Responded Unique Users / Survey Impression Unique Users. I finished both versions.

3. SQL Q2 — New vs. Tenured Users Average Rating

Schema:
survey(userid, date, survey_event, response)
users(userid, reg_date, country)

Q1. Compare the average survey rating of new users and tenured users.
My answer: The interviewer asked how I would define a new user. I defined new users as those who registered in the last 30 days and everyone else as tenured, and the interviewer accepted that. I used a CTE and CASE WHEN to define user_type, joined survey and users, then used AVG(CASE WHEN ...) to compute new_avg and tenured_avg separately.

Details from the room: in the second CTE I wrote WITH again, and the interviewer pointed it out and I fixed it. There was also one more syntax error that I never spotted, and it's not clear where it was.

Logistics: 45 minutes. The interviewer was about 4 minutes late, and my self-introduction took about 1 minute. Product Sense first, then SQL, and I only just finished both SQL questions.

I think the main reason I failed this interview is that I focused too much on the notification's own metrics and ignored the real business objective. The primary metric for the A/B test should have been the Customer Engagement Rate of the two groups, with vs. without notification, not notification CTR.

Also, my proposal of a 7-day window for CTR wasn't very reasonable. Usually if a user doesn't click a notification the day they receive it, they're unlikely to click it a few days later. Of course, the actual attribution window should still be decided from real data.

On top of that, if users can subscribe or unsubscribe from notifications on their own, then the Unsubscription Rate could also be an important metric for how popular the feature is.

Looking back now, I think the most fundamental problem was that the early discussion revolved too much around the notification itself, so my thinking stayed stuck on notification-level metrics and I never clarified the real product goal and business objective in time. Before choosing metrics, you should first pin down the real goal of the product, instead of diving straight into the specific metrics of one feature.

Feel free to leave comments with suggestions so we can all improve. Thanks!