Implement resilient LLM provider pool

Quick Overview

This English-language summary evaluates a candidate's ability to design a resilient, multi-provider LLM client library encompassing provider management, governance (rate limiting and quotas), resilience mechanisms (health checks, timeouts, circuit breakers), routing (cost- and latency-aware load balancing, retries and fallbacks), observability, security, and concurrency. Commonly asked in the ML System Design domain to assess architectural decision-making and trade-off reasoning for availability, cost, latency, and operational robustness, the question targets practical application and architectural-level thinking rather than purely conceptual or low-level implementation details.

Implement resilient LLM provider pool

Company: Perplexity

Role: Software Engineer

Category: ML System Design

Difficulty: hard

Interview Round: Technical Screen

Implement a resilient client library that routes requests across multiple external LLM providers. Support provider registration, capability mapping (models and context length), per-provider rate limiting and quotas, health checks, timeouts, and circuit breakers; cost- and latency-aware load balancing; retry and fallback strategies when a provider is degraded or down; observability (metrics, logs, tracing); secure key management; and thread-safe concurrency. Describe interfaces, data structures, and failure handling.

Overview: This English-language summary evaluates a candidate's ability to design a resilient, multi-provider LLM client library encompassing provider management, governance (rate limiting and quotas), resilience mechanisms (health checks, timeouts, circuit breakers), routing (cost- and latency-aware load balancing, retries and fallbacks), observability, security, and concurrency. Commonly asked in the ML System Design domain to assess architectural decision-making and trade-off reasoning for availability, cost, latency, and operational robustness, the question targets practical application and architectural-level thinking rather than purely conceptual or low-level implementation details.

|Home/ML System Design/Perplexity
Perplexity logo
Perplexity
Aug 13, 2025
hardSoftware EngineerTechnical ScreenML System Design
11
0

System Design Task: Resilient Multi‑Provider LLM Client Library

Context

You are designing a client library used by backend services to call external Large Language Model (LLM) providers (e.g., OpenAI, Anthropic, etc.). The library must route requests across multiple providers to maximize availability, control cost, and meet latency SLOs.

Requirements

  1. Provider management
    • Provider registration/unregistration
    • Capability mapping: available models, max context length, supported features (streaming, function calling, JSON mode, etc.)
  2. Governance
    • Per‑provider rate limiting (including per‑API key where applicable)
    • Quotas (per provider, per model, per tenant)
  3. Resilience
    • Health checks (active and passive)
    • Timeouts (connect, request, total)
    • Circuit breakers (fail‑fast, half‑open probes)
  4. Routing
    • Cost‑ and latency‑aware load balancing
    • Retry and fallback across providers when degraded or down
  5. Observability
    • Metrics, structured logs, and distributed tracing
  6. Security
    • Secure key management (storage, rotation, scoping, redaction)
  7. Concurrency
    • Thread‑safe, high‑throughput, supports parallel requests and streaming

Deliverables

  • Describe interfaces and data structures
  • Explain request routing, failure handling, and concurrency strategy
  • Include assumptions where needed

Submit Your Answer to Earn 20XP

Sign in to leave a comment

Loading comments...

Browse More Questions

More ML System Design•More Perplexity•More Software Engineer•Perplexity Software Engineer•Perplexity ML System Design•Software Engineer ML System Design