Design an Experimentation Platform: Assignment, Logging and Statistics for A/B Tests

You are asked to build the in-house system a product organization will use to run controlled experiments, such as A/B tests, on its website and mobile apps. How would you design it? ```hint Follow one user Trace a single user from the moment they open the app to the moment someone reads the test result: where their variant is decided, what is recorded about them, and where they are counted. ``` ### Clarifying Questions - Is this for one team running a few tests, or a company-wide platform where many teams experiment on the same users at the same time? - Must variants be decided on the device (for example in a mobile app that may be offline), on the server, or both? - Is the unit of randomization always a signed-in user, or must the system also handle logged-out visitors, devices or groups such as cities? - Should the platform only report results, or also recommend decisions and stop a test automatically when it harms key metrics? ### What a Strong Answer Covers - The platform's users and an experiment's lifecycle, from setup through ramp-up to a ship decision - Assignment that is random, consistent for each user across sessions, and safe when many experiments overlap - Exposure and event logging, and a metrics pipeline that reliably connects who saw what to what they did - A statistics layer covering sample size and duration, the right tests, and protection against peeking and multiple comparisons - Trust safeguards such as sample ratio checks, guardrail metrics, A/A tests and a kill switch ### Follow-up Questions - The observed traffic split in a test differs noticeably from the configured split. What do you check, and can you trust the result? - How would you test a change where treated users affect control users, such as a marketplace pricing change or a sharing feature? - How would you keep results trustworthy when hundreds of experiments run at once?