Run org-safe online experiment for recommender

Read the full interview experience this question came from →

Quick Overview

This question evaluates a data scientist's competency in online experimentation design, causal inference, clustered randomization, metric definition (primary, secondary, guardrails), sample size/MDE computation, variance reduction and sequential monitoring, instrumentation and privacy-aware logging.

Run org-safe online experiment for recommender

Company: Dropbox

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Technical Screen

Propose an online experimentation plan to evaluate the file recommender in production across multiple organizations where collaborators can influence one another. Specify: primary metrics (e.g., file open-through-rate, time-to-open), secondary/business metrics (productivity proxies), and guardrails (latency, error rate, privacy incidents, access denials). Choose the unit of randomization (org-, team-, or user-level) and justify to minimize spillover; describe bucketing, stickiness, and holdouts. Compute required sample size and MDE with clustering/ICC assumptions; select variance reduction (CUPED/stratification) and sequential monitoring approach with alpha spending. Detail ramp schedule, novelty and carryover controls, and interference detection. Define logging needed to reconstruct exposure and attribution, plus a difference-in-differences fallback if only partial randomization is possible. Explain stop/ship criteria and how to guard against Simpson’s paradox across tenants and roles.

Overview: This question evaluates a data scientist's competency in online experimentation design, causal inference, clustered randomization, metric definition (primary, secondary, guardrails), sample size/MDE computation, variance reduction and sequential monitoring, instrumentation and privacy-aware logging.

Read the full Dropbox Data Scientist interview experience this question came from

|Home/Analytics & Experimentation/Dropbox
Dropbox logo
Dropbox
Oct 13, 2025
mediumData ScientistTechnical ScreenAnalytics & Experimentation
6
0

Propose an online experimentation plan to evaluate the file recommender in production across multiple organizations where collaborators can influence one another. Specify: primary metrics (e.g., file open-through-rate, time-to-open), secondary/business metrics (productivity proxies), and guardrails (latency, error rate, privacy incidents, access denials). Choose the unit of randomization (org-, team-, or user-level) and justify to minimize spillover; describe bucketing, stickiness, and holdouts. Compute required sample size and MDE with clustering/ICC assumptions; select variance reduction (CUPED/stratification) and sequential monitoring approach with alpha spending. Detail ramp schedule, novelty and carryover controls, and interference detection. Define logging needed to reconstruct exposure and attribution, plus a difference-in-differences fallback if only partial randomization is possible. Explain stop/ship criteria and how to guard against Simpson’s paradox across tenants and roles.

Loading comments...