Design an IP Blocking Service Backed by a Slow, Unreliable Government API

Design a service that decides whether requests from an IP address must be blocked. Whether an IP should be blocked is known only to an external government API, which your service has to call. The interviewer added a set of requirements, including: - The government API's SLA is 1 second. - If the government API's failure rate is greater than 50% within 10 minutes, the system fails open. ```hint A second is long on a request path Decide which requests can avoid calling the government API at all, and what a request should do while a call for its IP is still in flight. ``` ```hint Make fail-open precise Define what counts as a failure, how a 10-minute rate is measured when many servers make calls, and what returns the system to normal checking. ``` ### Constraints and Clarifications - Reported: the government API's 1-second SLA and the fail-open rule above a 50% failure rate within 10 minutes. The interviewer gave further requirements of the same kind that were not reported. - Not given: request volume, the number of distinct IPs, how often the government's answers change, the API's rate limits, and what a request should do when a single call fails while the failure rate is under the threshold. Ask, or state your assumptions. - The round also included 10 to 15 minutes of behavioral questions. ### Clarifying Questions - Which traffic is checked: every request, or only sign-ins, new sessions and sensitive actions? - Does the 1-second SLA bound every response or a percentile, and may a request from a new IP wait that long? - How long may a government answer be reused, and should block and allow answers be reused for the same time? - Below the threshold, what does a single failed or timed-out call mean for its request: block, allow, or retry? - Does the government API limit the call rate, charge per call, or offer a bulk list? - Is the check per exact IPv4 or IPv6 address, or per address range? - Must each decision be auditable: which source decided, when, and in which mode? ### What a Strong Answer Covers - Settles what is checked, the latency budget, how long answers may be reused, and the behavior when one call fails - A decision cache with expiry and in-flight coalescing, so the 1-second API is off the path of almost every request - A precise failure-rate trigger over 10 minutes: what counts as a failure, how the rate is aggregated across servers, a minimum sample, and how the system recovers - Exactly what fail-open allows, and how decisions made while failing open are re-checked afterwards - Data model, audit trail, sizing, failure handling, and monitoring of the breaker state ### Follow-up Questions - The government adds an IP to its list. How soon is it blocked, and what bounds that delay? - During a fail-open period, an IP that should have been blocked gets through. How do you limit and detect the damage? - One server sees 80% of its calls fail while the fleet as a whole sees 30%. Does that server fail open? - The API accepts fewer calls per second than you have new IPs. What changes?