# Design an A/B Testing Platform Design a platform that lets product teams define an experiment, assign eligible users consistently to control or treatment, serve the assigned variant, collect exposure and outcome events, and compute trustworthy experiment results. The design should support multiple concurrent experiments and gradual traffic ramp-up. Focus on correctness of assignment and measurement as well as availability. You do not need to choose a particular cloud vendor or assume a numerical traffic target. ### Constraints & Assumptions - A user should normally remain in the same variant for the life of an experiment. - Assignment must honor eligibility rules and configured allocation percentages. - An outcome must be attributable only after a valid exposure. - Experiment configuration changes must be auditable. - The analytics path may be asynchronous; variant serving is latency-sensitive. ### Clarifying Questions to Ask - What is the unit of randomization: user, device, account, or request? - Must experiments be mutually exclusive within a product surface? - How fresh must metrics be, and which statistical decision rules are expected? - How should anonymous users, cross-device identity, bots, and late events be handled? - What failure behavior is safest when configuration or assignment services are unavailable? ### Solving Hints - Separate the online assignment path from the offline analysis path. - Make the assignment input and configuration version explicit in every exposure record. - Discuss sources of bias before discussing dashboards. ### What a Strong Answer Covers - Deterministic, unbiased bucketing and stable variant assignment. - Versioned experiment configuration, validation, approvals, rollout, and rollback. - Exposure logging with deduplication and linkage to downstream outcomes. - A streaming or batch pipeline that produces guardrail and primary metrics from reproducible data. - Interaction rules for overlapping experiments and mechanisms that prevent sample-ratio mismatch. - Monitoring for assignment failures, event loss, skew, stale configurations, and metric regressions. - Privacy, retention, access control, and auditability. ### Follow-up Questions 1. How would you change allocation from 1% to 50% without moving previously assigned users between variants? 2. How would you detect and diagnose a sample-ratio mismatch? 3. How would you support mutually exclusive experiments on the same surface? 4. What happens if an exposure event is delivered more than once or arrives after its outcome event?