Introduction to Load Balancers

What is load balancing?

A load balancer (LB) routes incoming traffic across a pool of eligible servers. Its policy decides which server receives a connection or request. The goal is to use available capacity and avoid sending work to unsuitable backends, not simply to give every server the same number of requests.

Design question: What signal tells the balancer that a backend can accept more work? Consider request cost, server capacity, and health.

Why do we need load balancers?

A load balancer can sit behind a firewall or at another boundary between service layers. As traffic and the backend fleet grow, it provides a common place to apply routing decisions:

• Scalability: Add or remove backends while keeping a stable client-facing endpoint. Define how new servers become ready and how old connections drain.

• Availability: Stop sending new traffic to servers that fail health criteria, provided healthy backends have enough capacity.

• Performance: Use a policy suited to the workload, such as connection count or measured response time, and evaluate whether it improves latency and utilization.

Here’s an abstract depiction of how load balancers work: