Production Hardening and Testing

What's being tested

Candidates must show they can take a working script or algorithm and make it production-ready: reliable HTTP interactions, safe retries, correct termination, observable failures, and testable behavior. Interviewers probe engineering judgment—tradeoffs around timeouts, concurrency, idempotency, and how you validate correctness with unit and integration tests. Expect follow-ups about edge cases (cycles, partial failures, rate limits) and how you instrument and roll out changes safely.

Core knowledge

• HTTP timeouts vs retries: set a sensible per-request timeout (e.g., 2–5s for interactive calls), separate from a global job deadline; never rely on TCP defaults. Use requests/aiohttp timeout params.

• Retry logic: retry transient 5xx/timeout/connection errors, but not 4xx (except 429); cap attempts and total retry window to avoid infinite loops.

• Exponential backoff with jitter: $t_n = base \times 2^{n}$ plus jitter (e.g., full jitter: sleep = rand(0, t_n)) to reduce thundering-herd.

• Idempotency: design operations so replays are safe—use an idempotency-key for POSTs or detect duplicate work via unique keys in Postgres/Redis.

• Circuit breaker and bulkhead: open circuit after X failures to avoid wasting resources, and isolate concurrent pools to prevent cascading failures.

• BFS/DFS & cycle detection: for traversal tasks, choose DFS for depth-first crawling, BFS for shortest-exit discovery; use a visited set or seen-bitset (memory: O(V)) to prevent infinite walks.

• Concurrency in Python: for IO-bound HTTP work prefer asyncio or thread pools; remember the Python GIL limits CPU-bound threads—use multiprocessing for CPU-heavy tasks.

• Rate limits & backpressure: honor Retry-After, implement token-bucket throttling, and bound in-flight requests to keep latency p95/p99 predictable.

• Observability & SLIs: emit latency histograms (p50,p95,p99), error-rate, retries, and queue depth; export to Prometheus and errors to Sentry.

• Testing pyramid: exercise business logic with fast unit tests (pytest) and HTTP contract tests using mocks (responses, unittest.mock) plus a few integration tests against a test server.

• Schema/contract validation: validate external responses with JSON Schema or pydantic models to fail fast and detect upstream changes.

• Deployment safety: use feature flags, small canaries, and CI/CD pipelines with linting, type checks (mypy), and test coverage gates.

Worked example — Find final URL by crawling until “congrats”

Start by clarifying: is link-following restricted to same domain, are redirects allowed, what response codes indicate transient vs permanent failure, and is there a max hop or time budget? A strong approach outlines three pillars: (1) a traversal loop that issues HTTP requests with a strict per-request timeout, (2) termination detection by inspecting response body for the literal "congrats" and a visited set for cycle detection, and (3) robust error handling: retry transient errors with exponential backoff + jitter, cap retries, and abort on repeated permanent errors. Implementation skeleton: synchronous or async request function, main loop that yields next URL(s), visited set check, and metrics/emitted events. Tradeoff to call out: use DFS (lower memory) vs BFS (finds shortest path quicker) depending on spec; picking asyncio increases throughput but adds complexity in tests and debugging. Close by saying you'd add unit tests for parsing and visited logic, integration tests against a local test server, and observability (retry counts, p99 latency); with more time you'd add canary runs and a dead-letter queue for unresolvable URLs.

A second angle — Find exit URL via BFS API calls

This shifts from HTML crawling to graph traversal over HTTP endpoints. Clarify node representation, neighbor listing API shape, and rate limits per API token. Use BFS with a queue and per-node visited set to guarantee shortest-path exit discovery; parallelize at the per-level granularity but bound concurrency to avoid bursting the API. Retries must be tuned for idempotent GET-like calls; if POSTs are used for traversal, insist on idempotency-keys. Also handle partial failures: persist frontier state to Redis or Postgres to resume after crashes. Emphasize measuring cumulative API calls (cost) and implementing circuit breakers or exponential backoff when many 5xx responses occur.

Common pitfalls

Pitfall: Retrying indefinitely on 4xx errors or non-idempotent operations.
Many candidates implement naive retries for every failure; instead, classify errors (4xx = client, 429 rate-limit special case, 5xx/transient) and only retry safe-to-retry classes with capped windows.

Pitfall: Forgetting global deadlines and resource bounding.
A loop with only per-request timeouts can still run forever. Always enforce a global job deadline and a max number of hops/requests to protect downstream systems and cost budgets.

Pitfall: Skipping tests for network code.
Saying “I’d test it later” is weak; prepare unit tests for parsing and state transitions, mocks for HTTP interactions (responses/unittest.mock), and a small integration harness that simulates latency, errors, and rate limits.

Connections

Interviewers often pivot to adjacent areas: they may ask about observability (how you'd alert on regressions), deployment strategy (canaries, rollbacks via CI/CD), or data-consistency concerns if traversal writes results to Postgres or a downstream queue. Be ready to explain storage choice, retries-on-write, and backfill strategies.

Further reading

• [Release It! by Michael Nygard] — practical patterns for circuit breakers, bulkheads, and production stability.

• AWS Architecture Blog: Exponential Backoff and Jitter — concise explanation and jitter strategies.