PracHub
QuestionsPremiumCoachesLearningGuidesInterview Prep
|Home/Analytics & Experimentation/Meta

Design an A/B test for WFH filter

Last updated: Mar 29, 2026

Quick Overview

This question evaluates proficiency in experimental design and causal inference for online products, including metric definition and attribution, randomization and identity handling, contamination control, power and duration calculations, variance reduction, and heterogeneity analysis.

  • hard
  • Meta
  • Analytics & Experimentation
  • Data Scientist

Design an A/B test for WFH filter

Company: Meta

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Technical Screen

A marketplace search page will add a displayed-but-optional 'Work from home' filter that, when clicked, narrows listings to remote-friendly options. Design an online A/B test to estimate the causal effect of showing this filter on conversion defined as bookings per visit (bookings/visits). Specify: (a) precise H0 and H1; (b) unit of randomization (e.g., visitor) and exposure/trigger definition (e.g., search-page load) including how returning users and cross-device identities are handled; (c) assignment and compliance rules (treatment sees the filter; control does not), including how to prevent cross-arm contamination; (d) metric definition: exact numerator/denominator, 28-day post-visit attribution window for bookings, tie-breaking when multiple visits occur before a booking, and whether to credit at the visit- or visitor-level; (e) guardrails (latency, bounce, listing CTR, cancellations) and pre-trend/AA/SRM checks; (f) segmentation by country and how to combine strata (e.g., inverse-variance weighting) and detect heterogeneity; (g) variance reduction (e.g., CUPED) and outlier/bot filtering; (h) sample size and duration to detect a 0.5pp absolute uplift from a 3.0% baseline at 80% power and alpha=0.05, including expected daily traffic; (i) analysis plan comparing ITT vs triggered analyses given only a subset click the filter, and how to estimate the effect among clickers without selection bias (e.g., instrumental variables or CACE); (j) risks (novelty, cannibalization, latency regressions, misattribution across visits) and mitigations.

Quick Answer: This question evaluates proficiency in experimental design and causal inference for online products, including metric definition and attribution, randomization and identity handling, contamination control, power and duration calculations, variance reduction, and heterogeneity analysis.

Related Interview Questions

  • Measure scheduled posts feature success - Meta (medium)
  • Estimate ads ranking revenue impact - Meta (medium)
  • How should you evaluate unconnected content? - Meta (medium)
  • Should WhatsApp launch group calls? - Meta (medium)
  • How would you grow Meta products? - Meta (medium)
Meta logo
Meta
Oct 13, 2025, 9:49 PM
Data Scientist
Technical Screen
Analytics & Experimentation
1
0

A/B Test Design: Optional "Work From Home" Filter on Search Page

You are designing an online controlled experiment for a marketplace search page that adds a displayed-but-optional "Work from home" (WFH) filter. When clicked, it narrows listings to remote-friendly options. The goal is to estimate the causal effect of showing this filter on conversion, defined as bookings per visit (bookings/visits).

Provide a complete test plan that specifies:

(a) Hypotheses

  • State the precise null (H0) and alternative (H1) hypotheses for the primary metric.

(b) Experimental Unit, Exposure/Trigger, Identity Handling

  • Choose the unit of randomization (e.g., visitor, household).
  • Define the exposure/trigger (e.g., search-page load) used to include observations in analysis.
  • Explain how returning users, logged-in status, cookies, and cross-device identities are handled (sticky assignment, identity stitching).

(c) Assignment and Compliance Rules

  • How users are assigned to treatment (filter visible) vs control (filter hidden).
  • Enforcement to ensure treatment sees the filter and control does not.
  • How to prevent cross-arm contamination (e.g., URL params, caching, shared devices).

(d) Metric Definition and Attribution

  • Precise numerator and denominator for the primary metric.
  • 28-day post-visit attribution window for bookings.
  • Tie-breaking when multiple visits occur before a booking.
  • Whether credit is assigned at the visit- or visitor-level (and any sensitivity alternative).

(e) Guardrails and Quality Checks

  • Guardrail metrics (e.g., latency, bounce rate, listing CTR, cancellations).
  • Pre-trend checks, A/A tests, and SRM (sample ratio mismatch) diagnostics.

(f) Country Segmentation and Aggregation

  • Plan to segment by country and combine strata (e.g., inverse-variance weighting).
  • How to detect and report heterogeneity across countries.

(g) Variance Reduction and Data Quality

  • Methods like CUPED/regression adjustment, stratification.
  • Outlier and bot filtering criteria.

(h) Power and Duration

  • Sample size and duration to detect a 0.5 percentage point absolute uplift (from 3.0% to 3.5%) at 80% power and alpha=0.05.
  • Include expected daily traffic assumptions and resulting duration scenarios.

(i) Analysis Plan: ITT vs Triggered vs Clickers

  • Compare intention-to-treat (ITT) vs triggered analyses given only a subset will click the filter.
  • How to estimate the effect among compliers/clickers without selection bias (e.g., instrumental variables/CACE).

(j) Risks and Mitigations

  • Identify key risks (novelty effects, cannibalization, latency regressions, misattribution across visits) and how to mitigate them.

Solution

Show

Submit Your Answer to Earn 20XP

Sign in to leave a comment

Loading comments...

Browse More Questions

More Analytics & Experimentation•More Meta•More Data Scientist•Meta Data Scientist•Meta Analytics & Experimentation•Data Scientist Analytics & Experimentation
PracHub

Master your tech interviews with 8,000+ real questions from top companies.

Product

  • Questions
  • Learning Tracks
  • Interview Guides
  • Resources
  • Premium
  • For Universities
  • Student Access

Browse

  • By Company
  • By Role
  • By Category
  • Topic Hubs
  • SQL Questions
  • Compare Platforms
  • Discord Community

Support

  • support@prachub.com
  • (916) 541-4762

Legal

  • Privacy Policy
  • Terms of Service
  • About Us

© 2026 PracHub. All rights reserved.