System Design: Real-Time Multi-Exchange Stock Data Ingestion and Live Web Page

Context

You are asked to design a production system that continuously ingests stock market data from multiple exchanges and renders an always up-to-date web page for users. Data includes trades and quotes (top-of-book), and the system must support historical queries and real-time streaming updates.

Assume: multiple heterogeneous data sources (direct exchange feeds and/or vendor APIs), variable market hours, exchange-specific schemas, intermittent network failures, and strict rate limits.

Requirements

Design the system and cover the following areas:

1. Data sources
   • Polling vs. streaming criteria and trade-offs
   • Connectors to multiple exchanges/vendors
2. Schema normalization
   • Canonical symbol mapping and data model for trades/quotes/book updates
3. Deduplication and idempotency
   • Event identity across venues/providers
4. Time handling and ordering
   • Time zones, trading sessions, and out-of-order events
5. Caching and storage
   • Hot cache, time-series storage, long-term archive
6. API design
   • Historical REST APIs and real-time streaming (WebSocket/SSE)
7. Client updates
   • Subscription, snapshot + delta model, resumability
8. Consistency and latency targets
   • SLAs/SLOs, read semantics
9. Fault tolerance and backfill
   • Retries, DLQs, replays, historical gap filling
10. Rate limiting
    • Upstream (exchanges) and downstream (clients)
11. Scaling and partitioning
    • Per-symbol/venue partitioning and horizontal scale
12. Observability
    • Metrics, logs, tracing, data quality checks
13. Security and compliance
    • Auth, transport security, secrets, market data entitlements
14. Cost considerations
    • Storage/compute trade-offs, data retention, egress

State assumptions where needed and justify design choices.