System Design: Weather Data Aggregation (10-minute Updates)

Context

Build a service that periodically fetches weather data from third-party providers for many cities/regions, stores and serves the data reliably, and scales with growth. Assume near–real-time use cases (dashboards, alerts, APIs) and historical analytics.

Requirements

Design and describe the following aspects:

1. Architecture and data flow end-to-end.
2. Scheduler that triggers updates every 10 minutes across many cities and providers.
3. Rate-limiting and quota management for third-party APIs.
4. Idempotency and deduplication across retries and providers.
5. Retry strategy, backoff, circuit breaking, and dead-letter handling.
6. Schema design for hot vs. cold storage and partitioning.
7. Caching strategy for read performance and cost control.
8. Data validation, normalization, and quality checks.
9. Multi-provider support, abstraction, and result blending/failover.
10. Handling provider outages and delayed/late-arriving data.
11. Monitoring, alerting, and SLOs.
12. Scalability across cities and regions, including capacity planning.

State any assumptions you need and justify key trade-offs.