## Scenario Design a healthcare data ingestion pipeline that can process millions of records efficiently. Records arrive from multiple partners in files and streaming batches, may contain sensitive data, may be duplicated or malformed, and can use different schema versions. Cover intake, validation, queue or broker selection, raw and curated storage, processing, fault tolerance, scaling, observability, replay, and handling of failed or unprocessed records. ### Constraints & Assumptions - Acknowledged input must remain replayable. - Transformations must be reproducible and versioned. - Access to sensitive fields is least-privilege and audited. - Delivery is at least once; deduplication uses a stable source record identity when available. - Clinical interpretation and regulatory policy are requirements to clarify, not assumptions to invent. ### Clarifying Questions to Ask - What formats, record rates, file sizes, latency targets, and retention rules apply? - Which source system provides a stable record or batch identifier? - Is ordering required per patient, encounter, or feed? - Which validation failures are rejectable, correctable, or quarantine-only? - Where may sensitive data be decrypted and who can replay it? ```hint Preserve before transforming Land an immutable encrypted copy and manifest before acknowledging. Downstream processors can then be retried or replaced without asking a partner to resend. ``` ### What a Strong Answer Covers - Authenticated intake, checksums, manifests, idempotent batch registration, and immutable raw storage. - A partitioned broker for record work, schema registry, versioned transformations, and deterministic deduplication. - Curated storage designed for stated access patterns, not one database for every use. - Quarantine and dead-letter flows with reason codes, replay tooling, and no sensitive payloads in ordinary logs. - Backpressure, autoscaling, disaster recovery, data-quality metrics, and raw-to-curated reconciliation. ### Follow-up Questions 1. How would you correct a transformation bug without duplicating downstream records? 2. What partition key balances throughput against per-entity ordering? 3. How do you prove that every accepted source record reached a terminal state?