Build a concurrent site crawler
Company: Salesforce
Role: Machine Learning Engineer
Category: Coding & Algorithms
Difficulty: medium
Interview Round: Onsite
Implement a small crawler in Python for a single target website.
Requirements:
- Input: a start URL, a maximum crawl depth, and an output CSV path.
- Crawl only pages within the same domain as the start URL.
- Use concurrency so that many pages can be fetched in parallel.
- For each successfully fetched HTML page, extract:
- page URL
- page title
- HTTP status code
- Deduplicate pages by normalized URL.
- Ignore non-HTML resources and skip pages that cannot be fetched.
- After crawling completes, sort the results by page title in ascending order and then by URL in ascending order.
- Export the final result set to a CSV file with columns: url, title, status_code.
Discuss the main edge cases and how you would improve this into a production-grade crawler.
Quick Answer: This question evaluates a candidate's ability to implement a concurrent web crawler, covering concurrency control, URL normalization and deduplication, HTTP handling, HTML parsing, and result serialization.