Design an **advertising pacing system** for a large-scale video / streaming advertising platform. Advertisers create campaigns with budgets, flight dates (start/end), targeting constraints, and delivery goals such as impressions, clicks, or spend. The platform receives ad opportunities (ad requests) in real time and, for each one, must decide which campaigns are eligible to serve — while **pacing delivery smoothly over the campaign lifetime** rather than exhausting the budget early. Your design should address how to: - Spread campaign spend / impressions over time instead of front-loading the budget. - Handle real-time ad requests with low latency. - Track spend, impressions, and remaining budget accurately. - Re-pace when traffic volume changes (diurnal cycles, spikes, dips). - Prevent overspending across many concurrent serving nodes. - Support nested budget scopes: campaign-level, ad-group-level, and daily budgets. Cover the data model, APIs, services, storage, streaming pipelines, and monitoring. Make the **separation between the fast serving path and the slower pacing control loop** explicit, and walk through requirements, end-to-end architecture, the pacing control loop, the real-time serving path, overspend prevention, event handling, scaling, and failure modes. ```hint Where to start The request path and the budgeting logic don't have the same latency budget. Think about which decisions *must* happen inside the per-request deadline and which can run on a slower cadence, and what state each side reads vs. writes. ``` ```hint Pacing signal Pacing is a feedback loop: define some notion of where a campaign's spend *should* be by now and compare it to where it actually is, then act on the gap. Ask yourself what makes that "should" target wrong when traffic isn't uniform over the flight, and what dials the serving path can actually turn in response. ``` ```hint Overspend under concurrency The dangerous case is many serving nodes deciding to spend the same campaign's budget at the same instant. Think about whether you really need a synchronous global check on every impression, or whether nodes can be handed something to spend locally — and what that choice does to your worst-case overspend. ``` ```hint Nested budgets & late events One serve can count against several budgets at once (ad-group, daily, lifetime, …). Think about what eligibility means when multiple caps apply, and separately about how delayed, out-of-order, or replayed spend events could corrupt your counters — and what property of event processing protects against that. ``` ### Constraints & Assumptions State your assumptions explicitly; a reasonable starting point: - **Scale:** ~1M+ ad requests/sec at peak across the fleet; hundreds of thousands of active campaigns; events (impressions/clicks/spend) at a similar rate to requests. - **Latency:** the per-request ad-decision budget is tight (single-digit to low tens of milliseconds, p99). Pacing and budget checks happen *inside* this budget. - **Accuracy:** hard budget overspend must be bounded (e.g. within a small single-digit percent worst case, ideally tighter); billing/spend ultimately reconciles against an authoritative ledger. - **Traffic is non-uniform:** opportunity volume varies by hour, day, region, and content — uniform "linear" pacing alone under- or over-delivers. - **Events can be late, out-of-order, or duplicated** (client retries, network, replay). ### Clarifying Questions to Ask - What is the delivery goal we're pacing to — spend, impressions, clicks, or conversions — and can a single campaign mix goals? - Is serving auction-based (we set a bid into a marketplace) or a direct/owned-inventory decision? This changes whether bid multipliers are even a lever. - What overspend tolerance is acceptable, and is the budget a hard cap or a soft target with reconciliation? - What's the relationship between the budget scopes (does an ad-group budget sum up to the campaign budget, or are they independent caps)? - What are the consistency requirements on spend tracking — is eventually-consistent accounting with later reconciliation acceptable, or must it be strongly consistent? - What's the expected number of candidate campaigns per request, and the p99 latency SLO for the ad decision? ### What a Strong Answer Covers - A clear **two-loop separation**: low-latency serving path vs. periodic pacing control loop, with which state each owns. - A pacing algorithm that goes **beyond linear** — forecast-aware target spend that adapts to non-uniform and shifting traffic, with a feedback signal that's stable (doesn't oscillate). - Concrete **throttling mechanisms** (admission probability, token bucket, bid multiplier) and *when* each is appropriate. - A credible **overspend-prevention** scheme that handles concurrent serving nodes (sharded counters / escrow tokens / two-level soft+hard enforcement) and names its overspend bound and tradeoffs. - Correct **nested-budget** handling (campaign / ad-group / daily) and how a serve must satisfy all applicable scopes. - An **event pipeline** that is idempotent and tolerant of late/duplicate/out-of-order events, plus reconciliation against authoritative billing. - **Data model, APIs, storage choices, caching strategy**, and how config/pacing state propagates to serving nodes. - **Scaling** (partitioning, hot-campaign handling), **monitoring** (delivery vs. target, overspend, pipeline lag, decision latency), and **failure modes** with explicit degradation behavior. - Articulation of the central **tradeoff**: smooth delivery vs. revenue vs. latency vs. strict budget correctness. ### Follow-up Questions - A single campaign is responsible for a large fraction of total spend and concentrates on one hot region — how do you pace it without that one counter becoming a serving bottleneck? - The traffic forecast is badly wrong for the next few hours (e.g. an unexpected live-event traffic spike). How does the controller react, and what protects you from over-delivering before the next control tick? - Mid-flight, an advertiser cuts a campaign's daily budget in half. How quickly and safely does that propagate to every serving node, and what prevents a brief window of overspend? - How would you extend this to **frequency capping** (per-user impression limits) on top of pacing, and how do the two interact when deciding eligibility?