Model preference without ground truth

Q: Model preference without ground truth

This question evaluates a data scientist's competency in uplift modeling, causal inference, experimental design, weak supervision, and bias and shift correction within the Machine Learning domain.

Q: How do I approach Machine Learning interview questions?

Machine Learning questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master machine learning interviews.

Q: What difficulty level is this interview question?

This is a hard difficulty Machine Learning question, commonly asked during Technical Screen rounds at Meta.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Question

Loading...

Problem: Designing an Uplift Modeling and Evaluation Strategy for Event Notifications Without Ground-Truth Labels

You need to decide which users should receive a new event notification, but you lack direct ground-truth labels of "appreciation." The business goal is to send notifications only when they create incremental value, not merely when a user is likely to click.

Assume:

You can randomize notification delivery during an exploration phase and log propensities.
You can track short- and long-horizon engagement and dissatisfaction signals (e.g., hides, unsubscribes).
Notification sending has a cost (e.g., user fatigue), and you want to optimize net benefit.

Answer the following:

Proxy labels and objective

Propose feasible proxy labels for "appreciation" (e.g., clicks, RSVP, downstream engagement), discuss pitfalls, and define an objective that targets incremental value (uplift) rather than propensity.

Data collection for identifiable counterfactuals

Design an exploration policy that logs propensities suitable for IPS/DR/SNIPS offline evaluation. Include guardrails to cap variance and protect user experience.

Offline metrics and online validation

Define offline metrics aligned with the business objective (e.g., policy value via inverse propensity weighting) and outline an online ramp plan to validate model quality.

Shift and bias correction

Explain how to detect and correct target shift and selection bias between exploration data and production (e.g., reweighting, domain adaptation).

Weak supervision and thresholding

If only weak signals exist, outline a weak-supervision or pairwise-preference approach and describe how you would calibrate the model and set decision thresholds.

Model preference without ground truth

Quick Overview

Problem: Designing an Uplift Modeling and Evaluation Strategy for Event Notifications Without Ground-Truth Labels

Solution

Comments (0)