AI-Paired Rate-Limited Wikipedia Crawler That Prioritizes Unseen Title Initials

This is a 75-minute assignment in which you pair with an AI coding assistant to build a crawler for Wikipedia. The round required using an assistant, such as an AI-enabled editor or a coding agent. The crawl order has a coverage rule: pages whose title starts with a first character that has not appeared yet are crawled first. For example, if the crawl starts from the page titled "Google", pages whose titles start with G are skipped for now. Once pages starting with all 26 letters have been crawled, crawl other pages at random. The crawler must respect a rate limit. ### Clarifying Questions - When does a letter count as covered: when a page with that initial is discovered, or when it has been fetched successfully? - How should titles whose first character is a digit, punctuation or a non-Latin letter be treated, and is the comparison case-insensitive? - What ends the crawl after all 26 letters are covered: a page budget, a time budget, or never? - How is the rate limit expressed (requests per second?), and does the server enforce it with error responses? - Should the crawler read article HTML, or may it use an API that returns a page's links? ### Part 1 — Crawl order with letter coverage Implement the crawler core: fetching a page, extracting links to other articles, and choosing the next page according to the coverage rule, then randomly once all letters are covered. ```hint Shape the frontier Think about how to organize the pages waiting to be crawled so that "any page with an uncovered initial" and "a uniformly random page" are both cheap to pick. ``` #### What This Part Should Cover - A frontier and visited-set design that supports the priority rule and the random phase - Normalizing links so the same article is not queued twice, and skipping non-article pages - What the crawler does when no page with an uncovered initial is currently known ### Part 2 — Respect the rate limit Make the crawler obey the rate limit and behave well when the server pushes back. ```hint Pace, then back off Separate steady pacing of requests from the reaction to an explicit "slow down" response. ``` #### What This Part Should Cover - A limiter that paces requests and can be tested without real waiting - Handling of rate-limit and transient error responses, including retries - Politeness beyond the limit itself ### Part 3 — Working with the assistant Explain how you split the work with the AI assistant during the 75 minutes, and how you keep the generated code understandable and correct. ```hint Own the design Decide which parts you specify yourself and how you check what comes back before building on it. ``` #### What This Part Should Cover - Planning the interfaces before asking for code - Reviewing and testing generated code in small, verifiable steps - Avoiding time lost understanding large blocks of unfamiliar generated code ### What a Strong Answer Covers - Clarified coverage, initial-character and stopping rules before coding - A working crawler whose ordering logic is testable without the network - Correct, testable rate limiting and back-off - Link normalization and deduplication - Deliberate, reviewable use of the AI assistant rather than accepting whatever it generates ### Follow-up Questions - Some letters, such as Q, X and Z, may take a long time to reach. How would you bound the effort spent looking for them? - How would you parallelize the crawler without exceeding the rate limit? - How would you make the crawl resumable after a crash?