How would you throttle a crawler?

The crawler above is now a production service. Expanding each page requires an outbound HTTP or API call to fetch its links, and upstream services may enforce strict rate limits. Design a crawler that: - Traverses a large set of pages or resources efficiently - Avoids per-host bottlenecks - Does not overwhelm upstream services or accidentally cause a denial-of-service - Handles retries, timeouts, and partial failures safely - Scales across multiple workers or machines Discuss the architecture, concurrency model, request scheduling, deduplication, rate limiting, backoff strategy, fairness across domains, and observability.