You have 3 variants, a churn guardrail, and delayed conversions. a) Design a Thompson Sampling bandit (specify likelihoods), and explain how you’ll handle delays (e.g., optimistic vs. debiased estimators) and non‑stationarity (e.g., discounting). b) Set traffic floors and fairness constraints. c) Define stopping and rollback policies when a guardrail is breached. d) Compare expected regret and business impact versus a fixed‑horizon A/B under seasonality.

This question evaluates proficiency in online experimentation and sequential decision-making, covering Bayesian bandits (Thompson Sampling), safety guardrails for churn, delayed-conversion handling, traffic allocation and stopping/rollback policy design.

How do I approach Machine Learning interview questions?

Machine Learning questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master machine learning interviews.

What difficulty level is this interview question?

This is a hard difficulty Machine Learning question, commonly asked during Onsite rounds at Meta.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Deploy multi-armed bandits safely | Meta Interview Question

Online bandit with 3 variants, churn guardrail, and delayed conversions

Context

You are running an online experiment with 3 variants (including control). The primary objective is to maximize conversions. There is a hard guardrail on churn: any increase in churn above a specified tolerance must trigger mitigation. Conversions are delayed relative to exposure.

Tasks

(a) Design a Thompson Sampling bandit:

Specify likelihoods and priors for both the primary metric and the churn guardrail.
Explain how you will handle delayed feedback (optimistic versus debiased estimators) and non-stationarity (e.g., discounting or sliding windows).

(b) Set traffic floors and fairness constraints across variants and key strata.

(d) Compare expected regret and business impact against a fixed-horizon A/B test under seasonality.

Assumptions to make explicit

3 variants include control.
Primary outcome is binary conversion within a defined window; churn is a binary event within a defined window; both can arrive with delay.
Guardrail tolerance is an absolute churn increase threshold g_max (could be set to 0 for no increase allowed).

Tasks

(a) Design a Thompson Sampling bandit:

Specify likelihoods and priors for both the primary metric and the churn guardrail.

Explain how you will handle delayed feedback (optimistic versus debiased estimators) and non-stationarity (e.g., discounting or sliding windows).

(b) Set traffic floors and fairness constraints across variants and key strata.

(d) Compare expected regret and business impact against a fixed-horizon A/B test under seasonality.

Deploy multi-armed bandits safely

Quick Overview

Deploy multi-armed bandits safely

Online bandit with 3 variants, churn guardrail, and delayed conversions

Context

Tasks

Assumptions to make explicit

Write your answer

Deploy multi-armed bandits safely

Quick Overview

Deploy multi-armed bandits safely

Online bandit with 3 variants, churn guardrail, and delayed conversions

Context

Tasks

Assumptions to make explicit

Write your answer