# Design a Billing-Token Rate Limiter for AI Agents Design a rate limiter that controls the number of billing tokens consumed by AI-agent requests. Requests have different estimated and actual token costs. Use a token-bucket model, explain where enforcement occurs, and handle concurrent requests, retries, long-running generations, and reconciliation when actual usage differs from the reservation. ### Constraints & Assumptions - Limits may exist at user, agent, and tenant levels. - The decision path must be fast enough to sit before model execution. - A request can stream output and discover its final token count only after it starts. - Repeated delivery of the same request identifier must not consume quota twice. ### Clarifying Questions to Ask - Are limits hard billing controls or soft abuse controls? - Which scopes must all pass, and how are their refill rates configured? - May a request exceed its initial estimate, and what should happen when it does? ```hint Reserve before work Charge an estimate atomically, then reconcile the difference when final usage is known. ``` ```hint Treat retry identity separately A transport retry should find the original reservation rather than create a second one. ``` ### What a Strong Answer Covers - Token-bucket state, refill arithmetic, and an atomic multi-scope reservation decision. - Reservation, commit, refund, timeout, and idempotency behavior. - Availability and consistency choices when limiter storage is degraded. - Metrics for denied tokens, reserved-versus-actual error, hot tenants, and reconciliation drift. ### Follow-up Questions - How would you support a short burst without allowing a tenant to exceed a daily budget? - How would you migrate a hot tenant without double refilling its bucket?