HTTP API Maze Crawling
Asked of: Software Engineer
Last updated

What's being tested
These problems test graph traversal over an implicit, HTTP-backed graph: each endpoint is a node, and its JSON-listed URLs are outgoing edges. You must combine BFS or DFS with URL normalization, cycle detection, bounded retries, timeout handling, and safe JSON parsing without confusing transient failures with a successful exit.
Patterns & templates
-
Use a
dequefor BFS when shortest hop count matters; use a stack for DFS when memory is constrained and any exit is acceptable. -
Maintain a
visited: set[str]and mark URLs when enqueuing; this prevents cycles and duplicate HTTP requests. -
Separate
fetch(url)from traversal: handle timeouts, retryable5xxresponses, invalid JSON, and non-success statuses in one place. -
Apply bounded exponential backoff, such as
min(base * 2**attempt, cap); never retry indefinitely or retry permanent4xxerrors. -
Resolve relative links with a URL parser such as
urljoin; normalize trailing slashes, fragments, and equivalent URL forms consistently. -
Stop immediately when the response contains the exact exit signal, such as a
"congrats"message; do not enqueue children from terminal nodes. -
Track complexity as
O(V + E)logical traversal work, plus network cost;visitedrequiresO(V)space and requests should be bounded.
Common pitfalls
Pitfall: Marking a URL visited only after processing its response allows duplicate queue entries and can amplify requests in graphs with converging paths.
Pitfall: Treating malformed JSON, timeouts, or
503responses as successful empty results can silently discard the only route to the exit.
Pitfall: Claiming BFS guarantees the fastest real-world result without clarifying that it minimizes graph hops, not latency or number of HTTP failures.
Practice these
The practice cards below cover the canonical variants — solve all of them and time yourself.
Practice questions
- Find the Exit of an HTTP URL Maze Despite Flaky 503s and Bad JSONRamp · Software Engineer · Technical Screen · medium
- Crawl an HTTP-served maze to its exit with 503 retries, dead ends and cyclesRamp · Software Engineer · Technical Screen · medium
- Find an Exit in a URL MazeRamp · Software Engineer · Technical Screen · medium
- Find final URL by crawling until “congrats”Ramp · Software Engineer · Technical Screen · hard
- Find exit URL via BFS API callsRamp · Software Engineer · Technical Screen · medium
- Navigate maze via HTTP APIRamp · Software Engineer · Technical Screen · medium
Related concepts
- HTTP API Crawling and URL MazesCoding & Algorithms
- Web Crawlers, URL Normalization, And PolitenessSystem Design
- Concurrent Web Crawlers and Work QueuesSystem Design
- RESTful API And HTTP Service DesignSoftware Engineering Fundamentals
- Graph Search, State Space, And Path OptimizationCoding & Algorithms
- Resilient API Aggregation And Operational DebuggingSoftware Engineering Fundamentals