# Debug and Improve a Load Balancer ### Clarifying Questions to Ask - Is the load balancer a single process exercise or a distributed production service? - What telemetry, configuration history, and backend test access are available? - Should policy state survive a load-balancer restart? ### Part 1: All Backends Appear Unhealthy Immediately after a configuration change, the load balancer marks every backend unhealthy and returns HTTP 500 even though direct requests to the service succeed. Walk through how you would isolate the fault. Include the health-check hostname and port configuration in your investigation. **Candidate hint:** Compare what the health checker actually sends with a known-good direct request before changing retry or routing logic. #### What This Part Should Cover - A disciplined narrowing process using logs, metrics, and a direct reproduction - Verification of target resolution, port, protocol, host header, and health path - Safe configuration correction and regression protection ### Part 2: One Backend Causes High Latency After health checks recover, aggregate latency remains high. One backend occasionally takes much longer than the others. Explain how you would instrument the request path, attribute latency to a backend, and distinguish queueing, network, and service time. **Candidate hint:** Decide which timestamps and labels are needed before deciding that the backend code is responsible. #### What This Part Should Cover - Per-attempt timing with backend identity and outcome - Tail-latency analysis rather than averages alone - A debugging sequence that avoids adding unbounded-cardinality telemetry ### Part 3: Add a Circuit Breaker Design a circuit breaker that stops sending normal traffic to a repeatedly failing backend while still allowing it to demonstrate recovery. Define states, transitions, thresholds, concurrency behavior, and interaction with existing health checks. **Candidate hint:** Specify who is allowed to send a probe after the cooldown and how simultaneous requests observe that decision. #### What This Part Should Cover - Closed, open, and half-open behavior with bounded probes - Failure classification, rolling evidence, and recovery criteria - Thread-safe state and protection against synchronized retries ### Part 4: Route by Observed Request Latency Replace round-robin selection with a policy that prefers backends with lower observed request latency. Explain the estimator, cold-start behavior, treatment of in-flight requests and failures, and how the policy avoids permanently starving a backend. **Candidate hint:** A raw lifetime average reacts slowly and can make an idle backend look better than a busy one. #### What This Part Should Cover - A stable, responsive latency estimate and sensible initialization - Load awareness, exploration, and failure penalties - Complexity, synchronization, and measurable rollout criteria ### What a Strong Answer Covers - Evidence-led root-cause analysis before code changes - Clear boundaries among health checking, passive failure detection, circuit breaking, and routing - Concurrency-safe state transitions and bounded failure amplification - An incremental rollout with tests, metrics, and rollback conditions ### Follow-up Questions 1. How do you avoid removing all backends during a shared dependency outage? 2. Which HTTP failures should count against a backend circuit? 3. How would retrying a request bias the latency estimator? 4. What changes when load balancers do not share state?