Design overload protection with load shedding evaluates requirements, scale assumptions, API/data design, architecture, trade-offs, failure modes, and rollout in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
Design a high-traffic service that maintains p99 latency SLOs under sudden spikes. Describe admission control, token-bucket rate limiting, priority queues, deadlines, and timeouts. Compare load shedding strategies (drop-new, drop-tail, random, deadline-aware) and when to apply each at the load balancer versus the application. Explain circuit breakers, backpressure to clients and queues, protecting critical dependencies, and the metrics/alerts you would track to validate effectiveness.
Quick Answer: Design overload protection with load shedding evaluates requirements, scale assumptions, API/data design, architecture, trade-offs, failure modes, and rollout in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
Design: Maintain p99 Latency SLOs During Sudden Traffic Spikes
Context
You are designing a user-facing, read-heavy HTTP/gRPC service that occasionally experiences sudden traffic spikes (for example, due to push notifications or viral content). The service must maintain a p99 latency SLO (e.g., 200 ms) and degrade gracefully under overload.
Assume a typical architecture: Clients → L7 load balancer/reverse proxy → stateless application instances → critical dependencies (cache, DB, search, feature store). Autoscaling cannot fully mask second-level spikes, so the service must protect itself.
Tasks
Admission Control and Rate Limiting
Describe how you would implement admission control at the load balancer and within the application.
Include token-bucket rate limiting (global and per-tenant/key), concurrency limits, and burst handling.
Queueing, Priorities, Deadlines, and Timeouts
Design request queues with priority classes (e.g., P0 interactive, P1 best-effort) and small, bounded backlogs.
Explain how you propagate and enforce request deadlines and set timeouts to meet the p99 SLO.
Load Shedding Strategies
Compare and contrast: drop-new, drop-tail, random (e.g., RED), and deadline-aware shedding.
Explain when each strategy is preferable, and where to apply it (load balancer vs. application).
Circuit Breakers and Backpressure
Explain circuit breakers for service-to-service calls (open/half-open/closed, trip conditions, fallbacks).
Describe how you provide backpressure to clients (HTTP/gRPC) and to internal queues.
Protecting Critical Dependencies
Show how you isolate and protect caches/DBs/search under overload (bulkheads, quotas, fallbacks, precomputed or cached responses).
Metrics and Alerts
Specify the metrics, SLI/SLOs, and alerting you would use to validate the effectiveness of your design under spikes.
Include brief numeric examples where helpful (e.g., choosing token-bucket parameters, concurrency caps, and queue budgets) and call out key trade-offs and pitfalls (retry storms, head-of-line blocking, etc.).
Clarifying Questions to Ask Guidance
Clarify users, core use cases, read/write patterns, scale, latency, availability, and data retention.
State explicit assumptions before making sizing or architecture decisions.
Prioritize the functional path first, then address reliability, security, observability, and rollout.
What a Strong Answer Covers Guidance
A scoped requirements summary with concrete non-goals and success metrics.
API, data model, architecture, consistency, capacity, and operations.
Reasoned trade-offs among simple and scalable designs, including bottlenecks and failure modes.
A validation, monitoring, migration, and launch plan appropriate for the risk level.
Follow-up Questions Guidance
What breaks first at 10x traffic or data volume?
How would you degrade gracefully during dependency failures?
What metrics and alerts would prove the design is healthy after launch?