Interview conceptCoding & Algorithms

HTTP API Maze Crawling

Asked of: Software Engineer

Last updated

Five-step BFS trace through an HTTP URL graph, showing normalization, visited tracking, bounded retries, cycle prevention, and stopping at a congrats response.

What's being tested

These problems test graph traversal over an implicit, HTTP-backed graph: each endpoint is a node, and its JSON-listed URLs are outgoing edges. You must combine BFS or DFS with URL normalization, cycle detection, bounded retries, timeout handling, and safe JSON parsing without confusing transient failures with a successful exit.

Patterns & templates

  • Use a deque for BFS when shortest hop count matters; use a stack for DFS when memory is constrained and any exit is acceptable.

  • Maintain a visited: set[str] and mark URLs when enqueuing; this prevents cycles and duplicate HTTP requests.

  • Separate fetch(url) from traversal: handle timeouts, retryable 5xx responses, invalid JSON, and non-success statuses in one place.

  • Apply bounded exponential backoff, such as min(base * 2**attempt, cap); never retry indefinitely or retry permanent 4xx errors.

  • Resolve relative links with a URL parser such as urljoin; normalize trailing slashes, fragments, and equivalent URL forms consistently.

  • Stop immediately when the response contains the exact exit signal, such as a "congrats" message; do not enqueue children from terminal nodes.

  • Track complexity as O(V + E) logical traversal work, plus network cost; visited requires O(V) space and requests should be bounded.

Common pitfalls

Pitfall: Marking a URL visited only after processing its response allows duplicate queue entries and can amplify requests in graphs with converging paths.

Pitfall: Treating malformed JSON, timeouts, or 503 responses as successful empty results can silently discard the only route to the exit.

Pitfall: Claiming BFS guarantees the fastest real-world result without clarifying that it minimizes graph hops, not latency or number of HTTP failures.

Practice these

The practice cards below cover the canonical variants — solve all of them and time yourself.

Practice questions

Related concepts