Experimentation Platform: Sync vs Async Validation, Latency Budgets and Health Checks

In a hiring-manager screen for a senior backend role close to experimentation infrastructure, you are asked for a technical deep dive into an online experimentation (A/B testing) platform. Instead of a full end-to-end design, the discussion concentrates on three areas: which validations should run synchronously and which asynchronously, how the platform stays inside the latency budget of the product services that call it, and which health checks tell you that both the platform and the experiments running on it can be trusted. For practice, assume a platform with the usual components: a control plane where experimenters create, edit and launch experiments (targeting rules, traffic allocation, variants, metrics); an assignment path that product services call while serving live requests to learn which variant a user should see; an exposure and event logging pipeline; and an analysis layer that turns those logs into metric results. Expect the hiring manager to keep asking why you placed each piece where you did. ### Constraints and Clarifications - The story does not say which kind of validation was meant. Treat it as covering both checks on experiment configuration (at create, edit and launch time) and checks on the logged exposure and metric data, and confirm the focus with the interviewer. - No latency figures were given. Ask for the callers' budget instead of assuming one, and express your design relative to it. - Every check or mechanism you propose should be tied to a concrete failure it prevents. ### Clarifying Questions - Is assignment computed by a remote service on each request, or inside the calling service by an SDK holding a cached copy of the configuration? - What share of the callers' latency budget may the experimentation call use, and is it measured at the median or at the tail? - When the platform cannot answer in time, should the caller serve the default experience, reuse a previous assignment, or fail the request? - Who is allowed to stop a misbehaving experiment: its owner, an on-call engineer, or automation? ### Part 1 — Synchronous versus asynchronous validation Decide which checks must block an action and which can run in the background. Cover the checks on experiment configuration when it is created, edited and launched, and the checks on exposure and metric data once traffic is flowing. For each check, explain what would go wrong if you moved it to the other side of the line. ```hint Price the mistake For each check, compare the damage of letting a bad experiment run for a few minutes with the cost of making an experimenter, or a live request, wait for the answer. ``` #### What This Part Should Cover - A clear rule for what blocks and what runs later, based on cost, determinism and the harm of missing a problem - Concrete examples of both kinds of check on the configuration side and on the data side - What happens when an asynchronous check fails: who is told, and whether the experiment pauses - Keeping validation work off the live assignment path ### Part 2 — Latency budgets Product services call the assignment path while they are serving users, so the platform spends part of their latency budget. Explain how you would keep assignment inside that budget at the tail, what a caller does when the platform is slow or unreachable, and how configuration changes reach the assignment path without adding latency to requests. ```hint Count the network hops Ask how much of an assignment decision truly needs a remote call, and what would have to be true for the decision to be computed locally and identically on every server. ``` #### What This Part Should Cover - Treating the budget as a tail-latency target rather than an average - Local or cached evaluation with deterministic assignment, and how configuration is distributed - Timeouts and a safe default that does not corrupt the experiment's data - Keeping exposure logging off the request path ### Part 3 — Health checks Describe the health checks you would run at two levels: the platform itself (is it up, and is it serving the configuration it should) and each running experiment (is its data trustworthy, and is the treatment harming users). Explain which checks should automatically pause or roll back an experiment and which should only alert. ```hint Up is not the same as healthy A service can answer every request while every experiment on it is producing biased results. Design checks that would catch each situation separately. ``` #### What This Part Should Cover - Basic liveness and readiness versus deeper checks such as configuration propagation and logging freshness - Data-trust checks such as traffic imbalance between arms and missing exposure records - Guardrail metrics and the conditions under which automation may stop an experiment - Avoiding false alarms caused by checking results over and over ### What a Strong Answer Covers - Ties every validation and health check to a specific failure it prevents - Keeps the live assignment path fast and simple and moves heavy work off it - Treats data trustworthiness as a health signal on par with uptime - Explains degraded behavior end to end, from a timeout in the caller to what the analysis later shows - Defends choices under pushback and says which decisions depend on information not yet known ### Follow-up Questions - An experiment reports a large, significant win, but the treatment arm received noticeably fewer users than configured. What do you do? - A configuration change with a broken targeting rule reaches half the fleet before anyone notices. How do you detect it and limit the damage? - How would you support an experiment on a backend path whose latency budget is far tighter than everyone else's? - Experimenters want to look at results every hour and stop as soon as they look significant. How should the platform respond?