Design a Reliable Third-Party Data Ingestion Service

# Design a Reliable Third-Party Data Ingestion Service Design a service that periodically pulls records from third-party providers and makes normalized data available to internal consumers. Providers may expose different APIs, quotas, pagination models, update semantics, and schemas. Focus first on correctness and recoverability. Then explain how the design scales as the number of providers, tenants, and records grows. ### Constraints & Assumptions - The source does not specify a domain, freshness target, or provider count; choose illustrative values and label them as assumptions. - A provider can time out, throttle requests, return partial pages, repeat records, or change a schema. - The system must avoid losing acknowledged records and must support replay after a bug. ### Clarifying Questions to Ask - Is ingestion scheduled, webhook-triggered, manually requested, or a combination? - What freshness, volume, retention, and tenant-isolation requirements apply? - Do providers expose stable IDs, cursors, modification timestamps, or deletion markers? - Must consumers see a transactionally consistent snapshot across an entire provider sync? ### What a Strong Answer Covers - Provider adapters, scheduling, durable work queues, raw capture, normalization, and a consumer-facing store. - Cursor and checkpoint ownership, idempotent writes, deduplication, retries, backoff, and dead-letter handling. - Protection against quota exhaustion, partial pagination, schema drift, and poison records. - Backfills, replay, observability, tenant isolation, secrets handling, and horizontal scaling. ### Follow-up Questions 1. How would you migrate a provider adapter when its response schema changes without warning? 2. How would you guarantee that a retry after a worker crash does not duplicate downstream records? 3. What changes if consumers need a complete point-in-time snapshot rather than eventually updated records?