Design a fault-tolerant load balancer for a cloud compute layer

Design a load balancer for the compute layer of a cloud platform: it receives client traffic and spreads it across a pool of backend compute instances. The interviewer's emphasis was on scaling to high load and on fault tolerance. The report gives no more detail, so settle the scope with the interviewer first. ```hint No single box The load balancer is itself a service that has to scale out and survive failures. Decide how traffic reaches one of many balancer instances in the first place. ``` ```hint Keep connections on their backend When a balancer instance dies or a new one joins, think about what happens to connections already in flight, and how another instance could choose the same backend for them. ``` ### Constraints and Clarifications - Not given: layer 4 or layer 7, traffic volume, the number of backends, and whether it is a shared multi-tenant service or one balancer per customer. Ask, or state your assumptions. ### Clarifying Questions - Is this a layer-4 balancer for TCP and UDP flows, or a layer-7 one that terminates TLS and routes on HTTP headers and paths? - Is it a multi-tenant service fronting many customers' pools of virtual machines, or internal to one service? - What throughput, number of concurrent connections and added latency must it support? - How quickly must traffic stop going to a failed backend, and must established connections survive the loss of a balancer instance? - Must backends see the client's original IP address? ### What a Strong Answer Covers - Scope (layer 4 or 7, tenancy), scale estimates and a latency budget - A scale-out data plane: how traffic is spread across balancer instances, then across backends - Backend selection and connection affinity that survive changes in balancer and backend membership - Health checking, draining and failure detection for backends and for balancer instances - A separate control plane for configuration, with safe rollout - Overload protection, tenant isolation and observability ### Follow-up Questions - A balancer instance dies. What happens to the TCP connections it was carrying? - One backend becomes slow but still passes its health checks. How do you detect it and shift traffic away? - One tenant's traffic spikes and threatens other tenants on the same balancer fleet. What do you do? - How do backends learn the client's original IP address, and what changes with direct server return?