Anthropic Software Engineer Interview Prep Guide

Focus most on caching/LRU, graph and grid traversal as they surface in crawler/frontier problems, partitioning basics, stateful stores, chat delivery, and Anthropic-flavored ML serving because you marked many related primitives shaky and have no solved-question signal yet. Merely review low-level kernel optimization and generic parsing/profiler work; your system design/fundamentals are 4/5 and many parsing, search, and transaction concepts are marked solid. Anthropic-specific highlights are GPU inference, model-weight activation, prompt systems, safety-evaluation pipelines, API quota enforcement, and mission-aligned leadership judgment rather than generic SaaS design. With one month, budget roughly 30 minutes for the Technical Screen sheet and 77 minutes for the Onsite sheet, then spend practice time on the emphasized cards.

Technical Screen — 30 min

System Design

• Concurrent Web Crawlers and Work Queues (Focus) — covered in depth under Onsite below.

• Chat System Design and Message Delivery (Focus) — covered in depth under Onsite below.

Coding & Algorithms

• LRU Cache Design and Canonical Keys (Focus) — covered in depth under Onsite below.

• Stateful In-Memory Ledgers and Versioned Stores (Focus) — covered in depth under Onsite below.

Onsite — 77 min

Behavioral & Leadership

Focus area — Anthropic screens mission judgment directly; with a 3/5 behavioral rating, prepare concrete safety trade-off stories.

What's being tested

Interviewers are probing judgment under ambiguity: how you prioritize safety trade-offs, own mistakes, and influence technical and cultural changes as a Software Engineer. Expect to demonstrate practical systems-level approaches to reduce risk, measurable outcomes you drove, clarity in trade-offs, and how you mentored or influenced others without overstepping. Anthropic cares because engineers build the runtime, observability, and guardrails that make AI systems safe in practice.

Core knowledge

• Risk quantification: express risk as expected harm: $\text{Risk} = P(\text{failure}) \times \text{severity}$; use rough orders-of-magnitude to prioritize mitigations when precise numbers are unavailable.

• Layered defenses: implement defense-in-depth: compile-time checks, runtime assertions, input validation, rate limiting, and a circuit breaker to contain unexpected behavior without single-point failures.

• Observability primitives: instrument via OpenTelemetry/Prometheus for metrics (error_rate, latency_p99), structured logs, and distributed traces; define alert SLOs and clear ownership for each alert.

• Safe deployment patterns: use canary rollouts, feature flags, progressive exposure, and automated rollback criteria (e.g., 3x baseline error_rate or latency_p99 increase) to limit blast radius.

• Testing strategy: combine unit tests, integration tests, property-based tests, and deterministic replay tests for critical paths; include fuzzing for unexpected inputs and adversarial examples at the interface layer.

• Post-incident process: run blameless postmortems with timeline, root causes, and action items; convert fixes into tests/monitoring and track completion in code reviews and PRs.

• Tradeoffs: latency vs safety: quantify added safety checks' cost (ms, CPU, dollars) and justify when to inline vs offload (e.g., async validation for non-blocking requests).

• Least privilege and access controls: enforce principle of least privilege across services and keys, rotate credentials, and log access attempts to make human/agent misuse auditable.

• Communication & influence: surface technical risk with concrete metrics and remediation plans; propose incremental changes (PR + testable rollout) — engineers influence by shipping defensible, reviewable artifacts.

• Ownership boundary: as a SWE, implement the systems, tests, and observability; defer product-level risk-benefit thresholds to PM/lead but provide clear technical recommendations backed by data.

• Measurable outcomes: specify targets (reduce error_rate by X%, cut mean time to detect MTTD to <Y minutes) and map each mitigation to measurable indicators.

• Escalation & decision rules: define explicit abort criteria and who can trigger emergency rollback; document them in runbooks and CI/CD playbooks.

Worked example — "Describe a Strongly Held View That Proved Wrong"

Frame the first 30 seconds: succinctly state the original position, context (project, scope, constraints), and why the view was reasonable (constraints, data, precedent). Clarifying questions: what stakeholders were impacted, what metrics measured success, and timeline for correction. Organize the answer into three pillars: (1) Evidence that overturned your view (logs, user metrics, incident timeline), (2) Actions you took to remediate and own the outcome (rollback, follow-up fixes, tests), (3) Lessons institutionalized (runbooks, new tests, team norms). Call out one technical tradeoff explicitly — e.g., you chose a fast inline validation for speed, but it increased latency and failed under load; you then moved to async validation with compensating checks. Close with accountability: describe how you communicated to stakeholders, tracked action items, and say "if I had more time, I'd add automated canary metrics and a replay test to prevent regression."

A second angle — "Discuss culture and mission alignment"

When asked about culture and mission alignment, translate mission language into tangible engineering practices: ship with clear safety SLOs, require safety-focused code review checklists, and ensure PR templates capture risk assessment and rollback plans. Emphasize mentorship: pair juniors on safety-critical diffs, run regular cross-team tabletop exercises, and maintain a rotating incident commander to distribute institutional knowledge. Show how you balance shipping velocity with mission by proposing measurable guardrails (e.g., every new surface must have a canary and a monitoring dashboard) and describe how you escalate unresolved trade-offs to leads with data-backed options.

Common pitfalls

Pitfall: claiming full ownership for cross-functional decisions.
Mistake: saying you “decided” a product-level safety threshold without involving PMs or legal. Better: describe how you recommended technical thresholds, provided data, and clarified whose decision it was.

Pitfall: vague mitigation actions without measurable follow-through.
Mistake: answering with "I fixed it" but not stating what tests, metrics, or dashboards prevented recurrence. Better: tie each remediation to a specific metric, test, or runbook.

Pitfall: over-technical or under-technical answers.
Mistake: dumping low-level details irrelevant to leadership judgment, or giving only high-level platitudes. Better: present a concise technical change plus its organizational impact and how you influenced adoption.

Connections

Interviewers may pivot to incident response and on-call practices, asking for a concrete runbook or escalation flow. They might also ask system-design safety tradeoffs (e.g., sandboxing vs. throughput) or for examples of mentoring and code-review process improvements that institutionalize safe behavior.

Further reading

• Concrete Problems in AI Safety (Amodei et al., 2016) — catalogs pragmatic failure modes and mitigation strategies relevant to engineering controls.

• Site Reliability Engineering (Google) — actionable practices for monitoring, SLOs, blameless postmortems, and incident management.

Practice questions

System Design

Focus area — Anthropic product context plus your API design, data modeling, search, permissions, and offline-sync interests justify deeper coverage.

What's being tested

Candidates must demonstrate end-to-end system design skills for a multi-tenant prompt playground: modeling metadata vs. large blobs, durable and consistent run records, low-latency execution and streaming, caching strategies, and operational concerns (storage costs, backup, observability). Interviewers probe tradeoffs between durability, latency, and cost; clear consistency boundaries; and pragmatic component choices you’d actually implement as a Software Engineer.

Core knowledge

• Metadata vs blob separation: store small indexed fields (owner, name, version, ACLs, tags, pointers) in `Postgres`/`CockroachDB`; store large prompt bodies and attachments in object store like `S3` or `GCS` with content-addressed keys (SHA-256).

• Content-addressable storage (CAS): use SHA-256 for deduplication; keep immutable blobs and a small metadata table mapping logical versions to blob keys; reference counting or GC tombstones for lifecycle.

• Versioning model: represent versions as immutable objects (commit id, parent pointer) and maintain a mutable head pointer for convenience; use optimistic concurrency (`ETag`/`version` column) for updates.

• Run records / provenance: write immutable run records to a durable store (append-only `Postgres` table or `Kafka` topic) containing prompt version, model adapter, runtime config, timestamps, and pointers to output blobs; ensure idempotent run submission with client-generated `run_id`.

• Streaming & durable execution: separate control plane (API) and execution plane (worker pool). Use gRPC or websockets for streaming tokens; persist intermediate outputs to ephemeral cache (`Redis`) and flush final output to `S3` + run record.

• Caching strategy: cache small, hot prompt templates and recent run outputs in `Redis`; for very large prompts, cache parsed/chunked representations and pre-warmed model-provider payloads; size-aware eviction (LRU with max-blob-size cutoff).

• Latency vs cost tradeoffs: cold fetch from `S3` adds tens to hundreds ms; prefetching and edge-caching reduce `p99` latency at higher storage/transfer cost. Quantify: if 1k requests/sec and average blob 1MB, bandwidth and egress costs dominate.

• Chunking & pagination: for very large prompts (>10s MB), chunk at storage time (e.g., 4–8MB) with index records so replay/streaming can fetch partial content; support range GETs to avoid reading entire blob.

• Consistency boundaries: enforce strong consistency for metadata (`Postgres` transactions), eventual consistency for blobs (object store achieves read-after-write for new keys in many providers; otherwise add verification), and causal links via run records referencing specific committed metadata version.

• Multi-tenant isolation & quotas: implement per-tenant namespaces for metadata keys and enforce read/write quotas at API gateways; use tenant-id in keys and in RBAC checks performed against the metadata DB.

• Security & privacy: encrypt blobs at rest with KMS; store sensitive fields (PII) in encrypted columns; implement audit logs for read/write and run execution; provide programmatic revocation by marking metadata versions revoked and enforcing at retrieval.

• Observability & SLOs: emit metrics: `create_prompt_latency`, `run_submission_p50/p99`, cache hit rate, `S3_get_latency` and error rates; trace end-to-end via distributed tracing (context through API → worker → provider).

Worked example — "Design An AI Playground For Very Large Prompts"

First 30 seconds: clarify scale (prompts size distribution, requests/sec, tenants, durability SLAs) and whether outputs must be immutable and reproducible. Assume multi-tenant, up to 1GB prompt sizes rarely, and reproducible runs required. Organize answer into three pillars: (1) storage/modeling (metadata DB + CAS blobs in `S3` with chunking), (2) execution model (API → durable queue → worker pool → streaming with ephemeral `Redis`), (3) correctness & ops (immutable run records, idempotency keys, observability). Flag an explicit tradeoff: storing full prompt in `Postgres` simplifies transactions but fails at scale and increases DB cost—prefer `S3` for blobs and keep only pointers in the DB. If time allows, add provider adapters (transformations, retries), background GC for orphaned blobs, and a migration plan for evolving schema.

A second angle — "Design a Prompt Sharing Product"

Here the core is similar but focus shifts to collaboration workflows, permissions, and safe execution. Model immutable prompt versions with provenance (author, parent, forks), and implement ACLs in the metadata layer for private/public visibility; use the same CAS blobs for storage. Add RBAC checks at read/write paths and ensure revocation semantics: marking a version revoked should prevent new runs and optionally delete blobs after legal hold checks. The sharing product also needs attribution metadata and immutable run records for auditability; streaming execution and caching strategies remain the same but with stricter access checks and possibly per-user encryption keys.

Common pitfalls

Pitfall: Treating the metadata database as a place to store large prompt text. This leads to poor performance, high storage costs, and long backup/restore times. Use object storage and keep metadata lean.

Pitfall: Assuming object stores have the same consistency semantics as transactional DBs. Don’t rely on eventual consistency for metadata references without verification; design transactions to write metadata pointing at a committed blob key.

Pitfall: Over-optimizing for `p99` without quantifying cost. Interviewers expect explicit tradeoff quantification (cache size vs `p99` gains vs egress/storage cost), not just "cache everything".

Connections

This topic often connects to provider adapters and model-serving interfaces, and to data-governance/audit systems (retention, deletion, legal hold). An interviewer might pivot to pipeline scaling (worker autoscaling, backpressure) or to secure multi-tenant key management.

Further reading

• [Designing Data-Intensive Applications — Martin Kleppmann] — deep treatments of storage, replication, and consistency tradeoffs.

• [AWS S3 Best Practices / Object Storage Patterns] — practical patterns for large-blob storage, chunking, and lifecycle (search "S3 multipart upload" in provider docs).

Practice questions

Focus area — You explicitly selected real-time messaging, WebSockets, offline sync, conflict handling, and fault tolerance.

What's being tested

Interviewers are checking practical mastery of designing a real-time, durable messaging pipeline that balances ordering, durability, and multi-device delivery under partial failures. Expect to demonstrate distributed-systems primitives (persistence, replication, partitioning), client sync protocols, and operational tradeoffs (latency vs durability, ordering vs throughput). Anthropic cares because chat is a microcosm of reliable, user-facing backend services requiring strong correctness, scalability, and clear tradeoff communication.

Core knowledge

• Message persistence: store canonical messages in a durable datastore (e.g., Postgres, Cassandra) with immutable IDs and a monotonic sort key; choose row-store for small scale, wide-column for high write fan-out and TTL requirements.

• Delivery vs storage separation: decouple write path (persist) from fan-out/delivery via a message queue like Kafka or SQS to provide backpressure, retries, and replayable offsets.

• Ordering models: per-conversation causal/total ordering vs per-sender ordering; implement per-conversation sequence numbers or Lamport clocks for causal ordering; enforce ordering within a partition (e.g., one Kafka partition per conversation).

• Idempotency: use client-supplied idempotency keys and server dedup index to guarantee at-most-once semantics for sends; follow Stripe-style idempotency patterns for retries.

• Multi-device delivery: maintain per-device delivery state and offsets; send messages via persistent WebSocket/gRPC streams for online devices and use push notifications for offline wake-up, with server-side replay on reconnect.

• Sync & reconciliation: store per-recipient read/recv receipts and last-seen offsets; on reconnect, client sends last-applied sequence number and server replies with messages > offset plus any membership changes.

• Failure & partial-write handling: prefer a write-ahead pattern: persist, emit to queue, acknowledge to client only after durable persist; use CDC to populate downstream indexes and delivery systems to avoid tight coupling.

• Scalability & partitioning: partition by conversation ID (hot-conversation mitigation by sharding sub-IDs or hashing with sticky routing); model throughput: if avg msg size S bytes and traffic T msgs/sec, bandwidth ≈ TS, and storage growth ≈ TS*retention.

• Consistency vs availability tradeoffs: for global low-latency, accept eventual delivery and reconcile via vector timestamps; for strict ordering across regions, prefer synchronous replication (higher p99s).

• Receipts & read-state: store receipts as compact metadata (per-user highest-seq or per-device set) to avoid per-message writes; for optional per-message receipts, amortize writes with batched updates to indexes.

Tip: keep the canonical message store authoritative and use change-data-capture to feed delivery pipelines and search/index services.

Worked example — Design a One-to-One Chat System

Frame quickly: ask about expected scale (messages/sec, messages/user), retention policy, ordering guarantees (per-conversation total order?), multi-device behavior, and whether receipts are required. Skeleton: (1) persist messages durably with immutable IDs and per-conversation sequence numbers; (2) emit to a queue (Kafka) for fan-out and replay; (3) deliver via per-device streaming connections (WebSocket/gRPC) and push notifications for offline devices; (4) sync on reconnect using last-seen sequence and reconciliation of membership/edits; (5) observability/ops: metrics for p99 delivery latency, consumer lag, and dead-letter queues. Flag a tradeoff: choosing single-partition per conversation simplifies ordering but limits throughput for extremely large groups or extremely hot one-to-one pairs; a sharded sequence or batching protocol can mitigate. Close by saying: if more time, detail schema (message payload, seq, idempotency key), partitioning plan, and sketches of failure scenarios (duplicate, reorder) with recovery protocols.

A second angle — Design a Resilient Chat System

With resilience and group chat emphasis, prioritize fan-out and membership-change handling: use an append-only canonical store plus a fan-out service that builds per-recipient delivery cursors, supporting idempotent replay. For groups, per-conversation ordering becomes harder; pick ordering semantics (per-sender or causal) and implement by assigning logical timestamps and using per-recipient queues to avoid global stalls. Membership changes require careful replay rules: new members should start at join time, removals stop delivery but require audit trails. Also emphasize monitoring consumer lag and automated repairs (rehydrate per-recipient cursors from canonical store when lag exceeds threshold).

Common pitfalls

Pitfall: A tempting design is to directly write to every recipient device synchronously; this blows up latency and availability when any recipient is slow or offline. Instead, persist first and fan-out asynchronously.

Pitfall: Assuming a single global sequence solves ordering; it creates a distributed bottleneck and complex leader election. Prefer per-conversation sequences or partitioned clocks.

Pitfall: Over-indexing per-message receipt writes for every delivery causes write amplification and costs; aggregate receipts into per-user highest-applied offsets or batch updates to reduce pressure.

Connections

This area naturally connects to stream processing and CDC (change-data-capture), mobile sync and conflict-resolution strategies, and observability for distributed systems (consumer lag, p99 delivery latency). Interviewers may pivot to rate-limiting, encryption/key management, or moderation pipelines.

Further reading

• Martin Kleppmann — Designing Data-Intensive Applications — chapters on replication, partitioning, and logs are directly applicable to chat systems.

• Apache Kafka documentation — Exactly-once semantics — practical patterns for durable emit-and-consume pipelines.

Practice questions

Focus area — You selected concurrency, synchronization, and distributed job scheduling; shaky graph/grid basics make crawler frontiers worth extra practice.

What's being tested

Candidates are evaluated on designing a concurrent web crawler that is correct, efficient, and robust: concurrency control for fetching, URL normalization and deduplication, polite per-host rate-limiting, frontier organization, and failure / retry behavior. Interviewers want to see the candidate ask the right scope questions (scale, single vs multi-domain, freshness), decompose into clear subsystems, and trade off practical choices (async vs threads, Bloom filters vs exact sets, centralized vs sharded frontier).

Core knowledge

• URL normalization / canonicalization: normalize scheme/host (lowercase), remove default ports, resolve relative paths, drop fragments, and canonicalize query params (sort or whitelist) to avoid combinatorial explosion from session IDs and tracking parameters.

• Same-domain / same-origin differences: same-domain may include subdomains; same-origin requires identical scheme+host+port. Clarify which constraint governs link acceptance and cookie/robot behavior.

• Frontier design: the URL frontier is a prioritized work queue; implement per-host queues with a global scheduler to enforce politeness and fairness (round-robin or weighted). For N up to ~10M, a single-machine in-memory frontier is OK; beyond that, shard by host to multiple machines.

• Duplicate detection: Bloom filter for large-scale membership with tunable false-positive p. Use $m = -n \ln p /(\ln 2)^2$ bits and $k = (m/n) \ln 2$ hashes; e.g., $n=100M$, $p=1e-6 \to m \approx 2.87e9$ bits (∼360MB).

• Concurrency models: choose between thread pool, async/await (aiohttp), or event-loop + worker processes. Async scales better for high I/O; thread pools are simpler when CPU-bound parsing dominates.

• Per-host politeness / rate-limiting: implement token-bucket or leaky-bucket per host and a global max_outstanding_per_host semaphore to avoid DOSing sites and respect robots.txt Crawl-delay.

• Retry / failure semantics: use lease/visibility timeouts (like SQS), exponential backoff for 5xx, idempotent storage for successful fetches, and a retry limit with a dead-letter queue for permanent errors.

• Cycle safety & depth control: maintain a visited set (or Bloom filter) and enforce max depth and per-domain URL caps to avoid infinite calendar or calendar-like traps.

• Politeness sources: parse robots.txt, honor crawl-delay, and respect Sitemap hints; cache DNS results and respect HTTP Retry-After.

• Storage & dedupe at content level: use content hashing (e.g., SHA-256) or canonical HTML signatures to detect duplicate pages with different URLs; store (url, content-hash, last-fetched) for freshness checks.

• Metrics & SLAs: track pages/sec, p99 fetch latency, queue depth, successes/failures, and per-host rate metrics; surface slow hosts and crawled-domain coverage.

• Scaling & sharding: shard by host hash to keep politeness local; coordinate frontier assignment via consistent hashing or a small master to avoid multi-master races.

Worked example — Design a Concurrent Domain Crawler

First 30s framing: clarify whether “domain” means exact hostname or includes subdomains, expected scale (pages/day), freshness requirements, and allowed content types (HTML only?). State assumptions: single logical domain, target 10M URLs, need politeness and breadth-first behavior. Organize the answer around four pillars: (1) frontier with per-host queue and global scheduler, (2) fetchers as async workers with per-host semaphores and token buckets, (3) deduplication using a Bloom filter for visited URLs plus content-hash dedupe, and (4) storage & retry with visibility leases and DLQ. Flag tradeoff: using Bloom filter saves memory but yields false positives (lost crawls); choose p based on acceptable misses and keep a small exact secondary store for recent URLs. Close by noting follow-ups: shard the frontier for higher scale, add politeness heuristics per subdomain, and instrument p95/p99 latency and coverage metrics for tuning.

A second angle — Crawl Same-Domain Links

This problem narrows to graph traversal inside one domain, emphasizing URL normalization, cycle-safe traversal, and depth constraints. The same core systems apply, but scale is smaller so you can use an exact Postgres visited table rather than probabilistic structures. Decide traversal strategy: BFS gives even coverage for site-mapping and search-indexing, while DFS uses less memory but risks deep traps. Here emphasize canonicalization (drop tracking params) and per-path heuristics to avoid calendaring traps (detect repeating numeric patterns). Rate-limiting and politeness are simpler (single host), so more budget can go to parsing and link extraction accuracy.

Common pitfalls

Pitfall: Underestimating duplicate-address space — assuming exact-string dedupe is sufficient will explode when query parameters or session IDs vary widely; always canonicalize and whitelist query parameters.

Pitfall: Not asking scope questions — failing to clarify single-domain vs multi-domain, scale, or freshness misses critical design constraints and leads to wrong architecture choices.

Pitfall: Overengineering concurrency — prematurely designing a distributed sharded system for a small crawl wastes time; start with async workers and per-host semaphores, then shard when throughput or memory demands justify it.

Connections

Interviewers often pivot to adjacent topics: designing a distributed task-queue with leasing semantics (visibility timeout, idempotent retries), or discussing content extraction/parsing performance and storage schema for crawled content. They may also shift into rate-limiting and backpressure strategies used broadly in distributed systems.

Further reading

• The Anatomy of a Large-Scale Hypertextual Web Search Engine (Brin & Page) — classic crawler/indexer architecture and tradeoffs.

• [Bloom Filters — Wikipedia / original references] — concise formulas and tradeoffs for probabilistic membership testing.

• Heritrix (Internet Archive crawler) docs — practical production crawler design and politeness implementation.

Practice questions

Focus area — You selected rate limiting and quotas; Anthropic API design needs fair sharing, abuse controls, and predictable tenant isolation.

What's being tested

Interviewers are probing your ability to design, implement, and operate robust API rate limiting and quota systems as part of a backend service. They want to see correct algorithm choice (token bucket/Leaky bucket/sliding window), distributed enforcement tradeoffs, client-server retry behaviour, observability, and how you reason about fairness and burstiness under production constraints. At Anthropic, this matters for protecting model-serving capacity, enforcing per-customer billing quotas, and preventing noisy neighbors from degrading latency for others.

Core knowledge

• Token bucket: tokens refill at rate r (tokens/sec); bucket capacity B allows bursts; request consumes tokens; tokens = min(B, tokens + r*Δt). Use for bursty, rate-limited APIs where sustained rate r is required.

• Leaky bucket vs token bucket: leaky bucket enforces constant output rate (smoothing), token bucket permits bursts then enforces average rate; pick token bucket for burst tolerance.

• Fixed-window vs sliding-window vs sliding-log: fixed-window is cheap but has boundary spikes; sliding-window percentile reduces spikes; sliding-log stores timestamps (accurate but heavy) or use time-window with approximate counters (e.g., two-window weighted).

• Distributed enforcement: centralized store (Redis) with atomic ops/Lua for counters or token counters offers consistency; local token buckets + periodic sync reduce latency but risk brief overcommit. Design around acceptable overage window.

• Strong vs eventual consistency: atomic INCR+TTL is simple but becomes a single point; sharded counters or CRDTs reduce contention at the cost of complexity and temporary over-allowance.

• HTTP semantics: return 429 Too Many Requests, include Retry-After header and human-readable quota/usage headers; provide per-key and global headers for client-side backoff.

• Retry/backoff: use exponential backoff with full jitter: base * 2^attempt (capped), then sleep = uniform(0, cap). Prevents thundering-herd coordination and reduces retry storms.

• Idempotency: for non-idempotent requests, require client-supplied idempotency keys (Stripe pattern) to safely retry when quotas or transient failures happen.

• Metrics and SLOs: instrument allowed, throttled, rejected, quota_exhausted counters and p50/p90/p99 latency; export to Prometheus and dashboards in Grafana. Track per-tenant throttling rate for billing/alerts.

• Quota models: per-second RPS, concurrent-inference slots, daily token budgets (e.g., tokens consumed by model calls). Convert diverse resources into a common cost metric where possible.

• Fairness & priority: implement weighted fairness or priority queues; avoid simple first-come first-served for multi-tenant fairness. Consider leaky-bucket per-tenant + global allocator.

• Throttling policies: soft limits with warning headers vs hard enforcement; consider grace periods for new customers and immediate hard limits for abuse.

• Protection patterns: circuit breakers, global caps, and slow-start ramps when a key first becomes active to guard cold-start spikes.

Worked example

Design a distributed per-customer rate limiter that supports bursts and a global capacity cap. First 30s framing: ask whether the limit is per-second or longer, whether bursts are allowed (size B), and whether overages must be strictly prevented or tolerable for short windows. Skeleton answer pillars: (1) choose token bucket per-customer with refill rate r and capacity B, (2) enforce centrally using Redis with Lua script for atomic token consumption and global counters, (3) add local caching (small local token-bucket) to reduce Redis calls and accept small overcommit with reconciliation. Tradeoff to flag: central Redis gives strong correctness but is a throughput bottleneck; local buckets improve latency but can transiently exceed global capacity—acceptable if overcommit window < few seconds and your billing/monitoring adjusts. For retries, specify exponential backoff with full jitter and return 429 + Retry-After. Close by saying: "If I had more time, I'd add borrow/stealing logic for idle tenants, quota-usage telemetry per customer, and automated alerts when global capacity approaches 80%."

A second angle

Consider enforcing both per-minute RPS and a monthly token quota (different resources). Frame it as a multi-dimensional quota problem: each request consumes a small instantaneous resource (RPS slot) and a long-term budget (tokens). Enforcement pillars: (1) check instantaneous token bucket/fixed-window counters for RPS and reject immediately if exceeded; (2) atomically decrement monthly token balance (stored in Redis or backed by a transactional store) and return quota_exhausted if depleted; (3) for scale, use a write-behind pattern where short-lived requests decrement a fast cache and reconcile with durable store asynchronously to avoid latency spikes. Tradeoffs: atomic cross-checks across dimensions raise latency; you may accept eventual reconciliation for the monthly quota with strict RPS enforcement to protect system.

Common pitfalls

Pitfall: Thinking a single algorithm fits all use cases. Token bucket, leaky bucket, fixed/window each target different workload shapes; choosing the wrong one leads to surprising bursts or excessive rejections.

Pitfall: Over-relying on local caches without bounding overcommit. A tempting optimization is full local enforcement to avoid Redis calls; without a reconciliation window this can exceed global capacity and break fairness.

Pitfall: Ignoring client-side behaviour and observability. Returning bare 429 without Retry-After, usage headers, or clear billing signals leads to poor client UX and repeated retries that amplify load; always provide actionable headers and instrument retry rates.

Connections

Interviewers may pivot to adjacent topics like API gateway scaling and placement (Envoy, Nginx), distributed coordination techniques for counters (consistent hashing, sharding), or billing/usage pipelines that consume quota telemetry. They might also ask about model-serving capacity planning and how rate limiting interacts with autoscaling.

Further reading

• Exponential Backoff And Jitter (AWS Architecture Blog) — practical patterns for resilient retries.

• How to Rate Limit an API (Cloudflare Engineering) — explanations of algorithms and production tradeoffs.

• Redis Rate Limiting Patterns (Redis Labs) — atomic counter/Lua recipes and implementation notes.

Practice questions

Focus area — You selected search/indexing and text matching; Anthropic prompts often require retrieval, context budgeting, provenance, and latency trade-offs.

What's being tested

Interviewers probe your ability to design a scalable, low-latency retrieval and prompt-assembly service that integrates many pieces: indexing, ranking, token accounting, caching, and operational tradeoffs. They want to see system-design instincts (sharding, replication, SLOs), algorithmic awareness (ANN vs exact search, multi-stage ranking), and pragmatic engineering decisions (batching, memoization, observability). The focus is on delivering correct, timely context to an LLM under real-world constraints (token limits, throughput, freshness).

Core knowledge

• Token budget math — compute available tokens: $T_{avail}=T_{context\_limit}-T_{expected\_response}-T_{prompt\_overhead}$; enforce with conservative safety margin (e.g., 10%).

• Chunking & overlap — split documents into chunks sized by tokens (e.g., 200–1000 tokens) with configurable overlap (10–30%) to preserve context across boundaries; too-large chunks reduce granularity, too-small increase retrieval noise.

• Vector representation & storage — embeddings dimensionality (e.g., 768–2048) drives memory: float32 costs ~4 bytes × dim per vector; quantization (PQ, OPQ) reduces memory/IO at recall cost. Tools: FAISS, Annoy, HNSW.

• ANN tradeoffs — Approximate Nearest Neighbor gives latency/throughput wins at recall cost; tune index parameters (ef, M for HNSW; nprobe for IVF) to balance p95 latency vs recall.

• Multi-stage retrieval — use a two-stage pipeline: cheap coarse ANN to get top-K, then expensive reranker (text-similarity or cross-encoder) on CPU/GPU for top-R (R ≪ K) to improve precision while containing cost.

• Sharding & replication — shard by document id or time window to scale memory/CPU; replicate shards for read SLOs and enable leader election for writes; consider consistent hashing to rebalance.

• Caching & memoization — cache frequent queries and assembled prompts (Redis or in-memory LRU). Cache key = (query fingerprint, schema version, prompt template hash). Invalidate on document updates using versioned keys.

• Latency SLOs & instrumentation — set p50/p95/p99 SLOs and measure each stage: embedding lookup, ANN query, rerank, tokenization, prompt assembly. Instrument with traces and per-component budget.

• Batching & concurrency — vectorization benefits from batching embeddings/reranking but introduces latency tail; use adaptive batching with max-wait and size thresholds to meet p95 SLOs.

• Tokenization and encoding effects — token counts depend on tokenizer (BPE/byte-level). Always measure tokenized size for chunk decisions and prompt assembly; do not approximate by character count.

• Freshness & consistency — choose update model: near-real-time incremental indexing vs periodic rebuilds. Incremental updates require write-path indexing and background index-merge; note transient inconsistencies during merges.

• Cost and IO profile — estimate RAM: vectors_count × dim × bytes; network IO for reranker results; GPU vs CPU cost for cross-encoders. Use quantized indices and caching to limit cloud spend.

Worked example — "Design a low-latency retrieval service that assembles a 50k-token context for LLM prompts"

First 30s: ask clarifying questions — required p95 latency (e.g., <300ms?), QPS, document corpus size, freshness window, expected response length, allowed cost. Declare assumptions: 10M documents, context limit 50k tokens, p95 500ms, near-real-time freshness (1–5m).

Skeleton of answer:

1. Ingestion & chunking: tokenize and chunk docs into ~1k-token chunks with 20% overlap; compute and store chunk metadata and embeddings in an index.

2. Index layer: use sharded HNSW/IVF+PQ indices for ANN; shard by doc-id ranges and replicate for reads.

3. Two-stage retrieval: ANN returns top-K (e.g., 200); cross-encoder reranks top-R (e.g., 20) on CPU/GPU; apply score thresholding.

4. Prompt assembly: tokenize selected chunks, apply token budget greedy selection (largest relevance-per-token first), enforce deduping and stable ordering; add template and safety placeholders.

5. Caching & batching: cache frequent query → assembled-prompt; batch embedding and rerank calls where latency allows.

One tradeoff to flag: pushing recall higher (bigger K, heavier reranker) improves quality but increases p95 latency and cost — tune to SLOs, possibly degrade gracefully (serve cached prompts under load). Close: "if time, add telemetry dashboards, A/B experiments on chunk size, and a background index compaction job."

A second angle — "Implement deterministic prompt assembly under high concurrency and document updates"

This variant emphasizes determinism and idempotency. Key changes: use versioned document IDs and a prompt-template hash in cache keys so identical inputs always yield identical prompts. Ensure stable ordering by sorting selected chunks by (document_version, chunk_offset) rather than dynamic scores alone; record selection signatures (checksums) to detect nondeterministic regressions. For concurrency, make index updates append-only with background compaction to avoid in-place mutability; use optimistic concurrency and publish index-version metadata so retrievals use a consistent snapshot. Here you trade some freshness and extra storage for deterministic behavior and easier debugging.

Common pitfalls

Pitfall: Ignoring tokenization mismatches — many engineers estimate tokens by characters; this causes silent prompt truncation or OOMs. Always measure tokenized tokens with the exact tokenizer used by the LLM.

Pitfall: Designing without SLOs or workload numbers — proposing heavy rerankers or huge K without throughput/latency constraints makes the design infeasible. Ask SLOs and target them explicitly.

Pitfall: Focusing only on embedding quality — embedding improvements matter, but production problems often come from missing caches, poor sharding, or no observability; prioritize operational robustness before micro-optimizing retrieval models.

Connections

This topic often leads to adjacent areas: distributed caching & consistency (cache invalidation strategies), index maintenance & compaction (background merges, offline rebuilds), and observability & SLO engineering (tracing, dashboards for p99 latency and recall metrics).

Further reading

• FAISS (Facebook Research) — GitHub — practical ANN implementations and quantization options.

• Efficient and Robust Approximate Nearest Neighbor Search using Hierarchical Navigable Small World Graphs (HNSW) — core paper explaining HNSW index behavior and parameters.

Practice questions

Coding & Algorithms

Focus area — You selected caching/eviction and marked LRU, LFU, Belady, write policies, and stampede handling shaky.

What's being tested

Candidates must show correctness and engineering rigor for an LRU cache: O(1) recency updates, deterministic memoization key construction across argument spelling/order, and safe persistence for crash resilience. Interviewers probe data-structure choice, canonicalization of function signatures, handling of non-hashable/nested inputs, and trade-offs between latency and durability.

Patterns & templates

• Doubly-linked list + hashmap — O(1) get/set/evict; implement nodes with key/value and move-to-front on access.

• collections.OrderedDict for quick Python prototype — popitem(last=False) for LRU eviction, O(1) average.

• Canonical signature binding with inspect.signature + Signature.bind — normalize positional/keyword into parameter-name order.

• Freeze supported values into immutable, type-tagged tuples (e.g., ("list", (..)), ("dict", (("k",v),..))) so lists/dicts are distinguished and hashable.

• Avoid caching exceptions — re-raise without storing failed results; only cache successful return values.

• Crash-resilience: append-only log or write-ahead log for mutations; persist both key→value mapping and recency order (or timestamps) and fsync at chosen durability points.

• Deterministic function identity — include function module + __qualname__ (or explicit id) in cache key to avoid cross-function collisions.

• Space/time tradeoff: persisting recency per operation increases p99 latency; batch checkpoints or async WAL flushes reduce latency but increase potential data loss.

Common pitfalls

Pitfall: Building keys from raw args/kwargs order — misses canonical binding and treats equivalent calls as different.

Pitfall: Using naive json.dumps for keys — loses type distinctions (tuple vs list) and can reorder dict keys unless sorted.

Practice these

The practice cards below cover the canonical variants — solve all of them and time yourself.

Practice questions

Focus area — You selected key-value stores, transactions, snapshots, concurrency control, quotas, and scheduling; practice progressive state-machine requirements.

What's being tested

These problems test implementing stateful in-memory ledgers and versioned stores: deterministic, time-ordered mutation application, per-key version history, and efficient historical reads. Interviewers probe correctness under ties/edge timestamps, read performance (historical snapshots), and simple concurrency/atomicity patterns a backend engineer should own.

Patterns & templates

• Event sourcing append-only log per-entity — store (timestamp, seq, op) and order by (ts, seq) for deterministic tie-breaking; append is O(1).

• Per-key history index: keep a vector or linked list per key and binary-search by timestamp for getBalanceAt(ts) in O(log m) where m is versions for that key.

• Tombstones & TTL: record delete markers with expiry metadata; treat tombstone as immutable state and purge lazily to avoid expensive synchronous GC.

• Atomic validation: implement applyEvent() as validate-then-commit using either per-key locks or optimistic CAS; ensure invariants (e.g., balance >= 0) hold before publishing.

• Prefix/range scans: keep keys in a sorted structure (std::map/B-tree) so prefix scans cost O(k + log n) and can iterate historical entries quickly.

• Scheduled jobs: schedule future payments in a time-priority queue (min-heap or calendar queue) and materialize them at execution time with idempotency keys.

• Merge/snapshot semantics: when merging accounts or applying promotions, snapshot rates/balances at effective timestamp to prevent retroactive changes to past reads.

Common pitfalls

Pitfall: assuming in-memory write order equals deterministic commit order — ties must be explicitly broken (timestamp+sequence).

Pitfall: scanning entire history per read — costly; use indexed per-key histories and binary search.

Pitfall: mutating past events (changing earlier ledger entries) instead of emitting compensating events or tombstones, which breaks reproducibility.

Practice these

The practice cards below cover the canonical variants — solve all of them and time yourself.

Practice questions

ML System Design

Focus area — Highly Anthropic-relevant; your selected API design, observability, fault tolerance, and quotas all apply to inference serving.

What's being tested

Interviewers test your ability to design a multi-tenant GPU-backed inference service that meets explicit latency SLOs while maximizing accelerator utilization and operational reliability. Expect to demonstrate end-to-end thinking: API contract and lifecycle, scheduling/admission control, batching and accelerator-level optimizations, model/version management, and observability for p99 latency and cost. Anthropic cares about pragmatic tradeoffs (latency vs throughput, isolation vs utilization) and clean, testable designs a Software Engineer can implement and operate.

Core knowledge

• API modes: synchronous streaming vs asynchronous / batch. Synchronous streaming (text-generation) needs low tail latency and chunked responses; asynchronous accepts job submissions and returns a job id for later polling or callback.

• SLO/SLI design: define p50, p95, p99 latency SLOs, throughput SLO, and error-rate; instrument request_latency, queue_time, inference_time, gpu_utilization. Use these to drive admission control and autoscaling.

• Queueing & admission control: model arrivals with $λ$, service rate $μ$ per GPU; utilization $ρ=λ/μ$. Use Erlang-C or M/M/c approximations to estimate queue wait $W$ and dimension capacity for a target p99.

• Batching math: throughput ≈ batch_size / batch_service_time; batch_service_time grows sublinearly in batch_size until memory/compute saturation. Optimize batch size to hit a target latency budget while maximizing throughput.

• GPU runtime constraints: CUDA context creation is expensive; memory fragmentation, model size, and kernel launch overheads limit effective concurrency. Use MIG, CUDA MPS, or container-level isolation for multi-tenancy tradeoffs.

• Model placement & sharding: small models fit on one GPU; large LLMs require tensor/model parallelism or pipeline parallelism across multiple GPUs, increasing inter-GPU communication (NVLink, NCCL) and latency.

• Dynamic batching & sequence batching: for autoregressive workloads, dynamic batching must respect token-level latency; implement sequence-aware batching that groups similar-length sequences and supports early flushing when latency budget is reached.

• Memory & input constraints: model + activation footprint must fit GPU memory; use quantization (int8, fp16) and offloading (CPU/RAM or NVMe) carefully: quantization reduces memory and latency but can alter quality.

• Scheduler & placement policies: implement a centralized scheduler (or per-cluster agent) that considers GPU memory, compute load, model residency, queue lengths, and tenant isolation; support pre-warmed models to avoid cold starts.

• Autoscaling & warm pools: scale at two levels: stateless worker replicas (if using model server) and GPU cluster capacity (node autoscaler). Maintain a warm-pool of preloaded models to avoid cold-starts for p99-sensitive paths.

• Failure modes & retries: detect OOM, driver resets, and preemption; isolate failures per request, use idempotency keys, and circuit-breakers to avoid retry storms that harm GPUs.

• Observability & debugging: correlate traces across API gateway -> scheduler -> model server -> GPU for a single request. Export gpu_memory, gpu_utilization, queue_depth, and batch_size_histogram for root-cause of p99 spikes.

• Cost & accounting: attribute GPU time per tenant/model using wall-clock and batch accounting. Consider spot/preemptible GPUs for non-SLO workloads and on-demand for SLO-critical paths.

Worked example — Design a GPU inference API

First 30 seconds: clarify SLOs (target p99 latency), supported API patterns (streaming vs non-streaming), concurrency characteristics, model sizes (single-GPU or multi-GPU), multi-tenancy/isolation requirements, and cost constraints. Skeleton pillars: (1) API contract & lifecycle (endpoints for POST /v1/generate synchronous, POST /v1/jobs asynchronous with job_id); (2) Request router & admission control that accepts/queues requests, enforces per-tenant rate limits, and computes batch eligibility; (3) Scheduler & runtime that places models into GPU resident pools, performs dynamic batching, and invokes Triton-style model servers; (4) Observability & autoscaling driving warm-pools and node scaling. A specific tradeoff: prioritize p99 by capping batch size and pre-warming models at the cost of lower throughput and higher GPU idle time; alternatively maximize utilization with larger batches but accept higher p99. Close: propose short experiments and metrics (simulate workloads, measure p99 vs throughput) and say "if more time, I'd design the exact admission-control policy, simulate it with traces, and prototype dynamic batching thresholds."

A second angle — Design a batch inference API

Batch inference is primarily asynchronous and throughput-first; design focuses on job lifecycle, chunking, checkpointing, and idempotent retries. Key changes: expose POST /v1/batches returning job_id, support resumable checkpoints for large datasets, and provide progress and partial-result streaming. Worker pool design uses a job queue with parallel workers that can aggregate multiple small inputs into GPU batches internally, but scheduling is simplified because strict p99 latency is relaxed. Emphasize durable storage for inputs/outputs (S3), rate-limits to avoid runaway costs, and cost-aware scheduling (spot instances for low-priority batches, guaranteed capacity for high-priority jobs). Also plan for per-job SLAs and shardability of large datasets to parallelize across GPUs.

Common pitfalls

Pitfall: Designing for average latency or throughput only. Many candidates tune batch sizes to maximize throughput and ignore queueing effects on p99, causing SLO misses under bursty traffic. Always dimension by tail latency and simulate with realistic variance.

Pitfall: Treating GPUs like infinitely shareable CPUs. Ignoring CUDA context costs, memory fragmentation, or multi-tenant interference leads to noisy neighbors and unpredictable p99 spikes. Explicitly model GPU residency and use hardware isolation primitives.

Pitfall: Over-architecting without measurable signals. A tempting deep design adds complex model-parallel pipelines; better answers first propose measurable experiments (traces, A/B of batching strategies) and incremental rollout plans with observability hooks.

Connections

Interviewers often pivot to adjacent topics: model rollout & canary deployments, feature-store/real-time retrieval for contextualized inference, or scheduling/resource management at cluster scale. Be ready to discuss how the inference design ties into CI/CD for models, monitoring regression in generation quality, and cost allocation per tenant.

Further reading

• NVIDIA Triton Inference Server docs — practical model-serving patterns and batching/runtime features.

• Ray Serve docs — patterns for scalable deployment and request routing for Python-first model servers.

• Google Borg paper — cluster scheduling principles useful for large-scale GPU placement and multi-tenant policies.

Practice questions

Focus area — Company-specific addendum: connect engineering design to safety evals, review workflows, metrics, and rollback decisions.

What's being tested

Interviewers are probing your ability to design and implement scalable, auditable red-team evaluation pipelines for large language models that reliably find, reproduce, and triage safety violations. They'll measure system-design skills: orchestration, throughput and cost tradeoffs, reproducibility, secure handling of harmful content, and integration with human-in-the-loop triage. Anthropic cares because engineering-quality pipelines let teams find safety regressions early and make mitigation work reproducible and measurable.

Core knowledge

• Red-team pipeline architecture patterns: ingestion → orchestration → execution → storage → triage; each stage needs retry semantics, idempotency, and strong audit logs to reproduce failures.

• Prompt corpus management: store canonical prompts, mutations, and metadata in `Postgres`/`BigQuery` with versioning and lineage; use immutable IDs to map runs to artifacts.

• Orchestration tools: prefer `kubernetes` + job queue (or `Argo`/`Airflow`) for reliability; use queues (`Kafka`, Redis streams) for backpressure and exactly-once semantics where possible.

• Execution isolation: run evaluations in containerized sandboxes (`Docker`) with strict resource limits, network egress controls, and per-run ephemeral credentials to avoid leaking secrets or sensitive outputs.

• Human-in-the-loop triage: build a triage UI that surfaces context (prompt, model version, config, seed, reproducible run ID) and supports labeling/priority and escalation; persist labels in `Postgres` with audit trails.

• Reproducibility primitives: record RNG seeds, model commit hashes, tokenizer versions, and full runtime configs; attach a stable repro ID to every execution for exact replay.

• Safety of outputs: treat model outputs as sensitive data; redact PII automatically (PII detection pipeline) before storage, encrypt at rest, and restrict access using `Vault`/IAM; maintain deletion and retention policies.

• Metrics & SLOs: track throughput (QPS), latency (`p95`, `p99`), failure rate, time-to-triage, triage backlog, and precision/recall of automated detectors; define SLOs for evaluation job success and triage SLAs.

• Sampling & statistical considerations: use stratified sampling across prompt types, steerable priors for rare harms, and compute required sample sizes for target error bounds; be wary of survivorship bias from filtered corpora.

• Cost & capacity planning: estimate workers = ceil((QPS * avg_latency) / concurrency_per_worker); optimize by batching inferences, using smaller models for fast fuzzing, and GPU autoscaling rules.

• Automated detection vs. human labeling: combine lightweight classifiers for high-recall filtering and humans for high-precision adjudication; record detector confidence and use for active learning.

• Security & compliance: log audit trails for access, use role-based access controls, and isolate red-team outputs from production logs to prevent accidental release.

Worked example — "Design a scalable red-team pipeline for LLM safety evaluation"

First 30 seconds: ask clarifying questions — expected daily evaluation volume, whether evaluation runs against offline checkpoints or live-serving endpoints, required replay fidelity, and acceptable latency/cost tradeoffs. Skeleton answer pillars: (1) ingestion and corpus versioning (immutable IDs, `Postgres`/`S3` storage), (2) orchestration and execution (queueing with `Kafka`/K8s jobs, sandboxed containers), (3) detection and triage (automated detectors + human UI), and (4) observability/alerts (metrics, logs, reproducibility). A concrete tradeoff to flag: running full red-team on the largest checkpoint vs a cheaper proxy model for broad fuzzing — cheaper proxies increase throughput but may miss model-specific behaviors. Implementation detail to call out: store a reproducible artifact bundle (prompt, seed, model hash, tokenizer) per failing example to enable deterministic replay. Close by saying: "If I had more time I'd prototype a small end-to-end run, instrument `p95` latency and triage throughput, and build a simple human UI to validate workflow assumptions."

A second angle — "Implement automated safety checks for model outputs in production"

Same principles apply but constraints shift: latency and availability matter more, and you must avoid blocking healthy traffic. Use a two-tier approach — synchronous lightweight checks for immediate blocking (fast regexes, tiny classifiers), and asynchronous deep checks routed to the red-team pipeline for richer analysis. Prioritize lightweight detection rules for high-precision blocking; funnel uncertain cases into offline evaluation with reproducible IDs. Also add feature-flagged canaries and gradual rollout to detect false positives quickly. The engineering emphasis moves from batch throughput to low-latency inference, cache-friendly detectors, and stricter access controls to protect user data.

Common pitfalls

Pitfall: assuming automated detectors are ground truth — over-reliance leads to large false-positive/negative budgets and wasted triage cycles. Always pair with human adjudication and track detector precision/recall.

Pitfall: inadequate reproducibility metadata — failing to record model commit, tokenizer, and seed makes triage impossible; every failing example must include a reproducible artifact bundle.

Pitfall: designing for average-case throughput only — neglecting burst load and backpressure causes queue saturation and lost runs; design autoscaling and durable queues (`Kafka`, SQS) with dead-letter handling.

Connections

• Observability & incident response: tie red-team alerts into on-call tools and runbook automation so safety regressions trigger the same operational rigor as outages.

• Model deployment & canarying: pipelines often integrate with deployment flows (canary, shadowing) to surface safety regressions before full rollouts.

• Data governance & privacy: red-team outputs frequently include sensitive data, so coordinate with compliance and apply PII redaction, retention, and access controls.

Further reading

• The Twelve-Factor App — operational design patterns useful for building reproducible evaluation services.

• [RFC-style runbook patterns (examples)] — search for "reproducible experiment artifact" guides to standardize artifact capture across teams.

Practice questions