Implement a Rate Limiter for CI Jobs Rather Than for API Requests

In a technical screen built around your experience with cloud infrastructure, Terraform and CI/CD, the interviewer asks how you would implement a rate limiter. The interviewer does not mean rate limiting an HTTP API at the edge, for example with a managed web application firewall rule; pointing to such a product does not answer the question. The goal is a rate limiter for jobs inside a CI system: explain how you would implement it, and write the core limiting logic. ### Clarifying Questions - What does the limit protect: a shared resource that CI jobs call (an artifact registry, a cloud provider's API, a deployment target), the runner fleet itself, or fairness between teams? - Is the limit on how many jobs start per unit of time, on how many run at once, or both? - What should happen to a job over the limit: wait in a queue, be delayed and retried, or fail with a clear message? - Is the limit global, or per repository, per team or per target environment? - Do jobs run on many runner hosts, so that the limiter's state must be shared? ### Part 1 — Choose and implement the limiting algorithm Choose an algorithm that enforces a sustained rate of job starts while allowing a bounded burst, and implement it as a small class that a scheduler can call before starting a job. ```hint Bursts are normal in CI Many pipelines start together after a merge or at a scheduled time. Decide how the limiter should treat a burst that fits within the long-run budget, and how a caller learns how long to wait when it does not fit. ``` #### What This Part Should Cover - A comparison of the standard algorithms and a justified choice for CI workloads - A correct, thread-safe implementation whose clock cannot run backwards - What the limiter tells a job that is over the limit, and the cost of each check ### Part 2 — Run it across a distributed CI system Jobs are scheduled by a controller and run on many runner hosts. Explain where the limit is enforced, where its state lives, and how the system behaves when parts of it fail. ```hint One budget, many runners A limiter inside each runner sees only its own jobs. Decide where the shared state lives, how a check and its update happen atomically, and what a job does while it waits. ``` #### What This Part Should Cover - The enforcement point (scheduler admission or a step inside the job) and the shared, atomic state behind it - Waiting versus failing for over-limit jobs, and fairness between teams - Behaviour when the shared store is unavailable, and how limits are configured and observed ### What a Strong Answer Covers - Recognizes that the question is about CI jobs, not API traffic, and clarifies what the limit protects - A justified algorithm choice that handles bursts, with correct implementation details - Atomic shared state across runners, without relying on the runners' clocks agreeing - Over-limit jobs delayed with a known wait time rather than failing arbitrarily - Failure behaviour, fairness, configuration and observability ### Follow-up Questions - The shared store holding the limiter state becomes unreachable. Should CI jobs proceed or wait, and does the answer depend on what is being protected? - One team's scheduled pipelines use the whole budget every night. How do you make the limiter fair across teams? - How would you also cap the number of jobs running at once, and how does that differ from a rate limit? - How do you choose the burst size and the sustained rate, and change them without redeploying the runners?