You are asked to predict which users would appreciate the new event notification, but you lack direct ground-truth labels. 1) Propose feasible proxy labels and their pitfalls (e.g., clicks vs RSVP vs downstream engagement), and define an objective that targets incremental value (uplift) rather than propensity. 2) Design a data-collection strategy that yields identifiable counterfactuals: specify an exploration policy with logged propensities suitable for IPS/DR/SNIPS offline evaluation; describe guardrails to cap variance. 3) Define offline metrics aligned with the business objective (e.g., policy value via inverse propensity weighting) and an online ramp plan to validate model quality. 4) Explain how you would detect and correct target shift and selection bias between exploration data and production (e.g., reweighting, domain adaptation). 5) If only weak signals exist, outline a weak-supervision or pairwise-preference approach and how you would calibrate the model for decision thresholds.

This question evaluates a data scientist's competency in uplift modeling, causal inference, experimental design, weak supervision, and bias and shift correction within the Machine Learning domain.

How do I approach Machine Learning interview questions?

Machine Learning questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master machine learning interviews.

What difficulty level is this interview question?

This is a hard difficulty Machine Learning question, commonly asked during Technical Screen rounds at Meta.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Model preference without ground truth | Meta Interview Question

Problem: Designing an Uplift Modeling and Evaluation Strategy for Event Notifications Without Ground-Truth Labels

You need to decide which users should receive a new event notification, but you lack direct ground-truth labels of "appreciation." The business goal is to send notifications only when they create incremental value, not merely when a user is likely to click.

Assume:

You can randomize notification delivery during an exploration phase and log propensities.
You can track short- and long-horizon engagement and dissatisfaction signals (e.g., hides, unsubscribes).
Notification sending has a cost (e.g., user fatigue), and you want to optimize net benefit.

Answer the following:

Proxy labels and objective

Propose feasible proxy labels for "appreciation" (e.g., clicks, RSVP, downstream engagement), discuss pitfalls, and define an objective that targets incremental value (uplift) rather than propensity.

Data collection for identifiable counterfactuals

Design an exploration policy that logs propensities suitable for IPS/DR/SNIPS offline evaluation. Include guardrails to cap variance and protect user experience.

Offline metrics and online validation

Define offline metrics aligned with the business objective (e.g., policy value via inverse propensity weighting) and outline an online ramp plan to validate model quality.

Shift and bias correction

Explain how to detect and correct target shift and selection bias between exploration data and production (e.g., reweighting, domain adaptation).

Weak supervision and thresholding

If only weak signals exist, outline a weak-supervision or pairwise-preference approach and describe how you would calibrate the model and set decision thresholds.

Problem: Designing an Uplift Modeling and Evaluation Strategy for Event Notifications Without Ground-Truth Labels

Assume:

You can randomize notification delivery during an exploration phase and log propensities.

You can track short- and long-horizon engagement and dissatisfaction signals (e.g., hides, unsubscribes).

Notification sending has a cost (e.g., user fatigue), and you want to optimize net benefit.

Answer the following:

Proxy labels and objective

Propose feasible proxy labels for "appreciation" (e.g., clicks, RSVP, downstream engagement), discuss pitfalls, and define an objective that targets incremental value (uplift) rather than propensity.

Data collection for identifiable counterfactuals

Design an exploration policy that logs propensities suitable for IPS/DR/SNIPS offline evaluation. Include guardrails to cap variance and protect user experience.

Offline metrics and online validation

Define offline metrics aligned with the business objective (e.g., policy value via inverse propensity weighting) and outline an online ramp plan to validate model quality.

Shift and bias correction

Explain how to detect and correct target shift and selection bias between exploration data and production (e.g., reweighting, domain adaptation).

Weak supervision and thresholding

If only weak signals exist, outline a weak-supervision or pairwise-preference approach and describe how you would calibrate the model and set decision thresholds.

Model preference without ground truth

Quick Overview

Problem: Designing an Uplift Modeling and Evaluation Strategy for Event Notifications Without Ground-Truth Labels

Solution

Submit Your Answer to Earn 20XP

Model preference without ground truth

Quick Overview

Problem: Designing an Uplift Modeling and Evaluation Strategy for Event Notifications Without Ground-Truth Labels

Solution

Submit Your Answer to Earn 20XP