# Design Work Orchestration for Machine-Learning Data Pipelines Design an orchestration system for data-production workflows used by machine-learning teams. A workflow is a directed acyclic graph of tasks that may ingest, transform, validate, and publish versioned datasets. Users submit workflows, trigger scheduled or ad hoc runs, observe task status, retry failures, and trace each published artifact back to its inputs and code/configuration version. ### Constraints & Assumptions - Tasks can run for minutes or hours and may execute on different compute pools. - Workers and orchestrator processes can fail, retry, or send duplicate updates. - A run must not publish a dataset unless its required validations succeed. - Reproducibility and lineage matter in addition to throughput. - Task side effects are not generally exactly once; idempotency must be designed explicitly. ### Clarifying Questions to Ask - Are workflows batch-only, or must event-driven triggers be supported? - What task runtimes and compute backends are in scope? - Which artifacts require immutable versioning and retention? - How are secrets, tenant isolation, and data-access permissions enforced? ### Part 1: Control-Plane Model Define workflow versions, run and task-attempt states, dependencies, artifacts, and user APIs. #### What This Part Should Cover - Immutable workflow/run identity and validated DAG structure. - Separate logical task instances from retry attempts. - Artifact lineage binding inputs, outputs, code, configuration, and validation results. ### Part 2: Scheduling and Execution Explain readiness evaluation, queueing, worker leases, retries, cancellation, and safe publication. #### What This Part Should Cover - Durable state transitions and idempotent worker protocols. - Resource-aware scheduling across compute pools with quotas and backpressure. - Staging plus atomic publication only after validation. - Recovery from lost workers and duplicate completion messages. ### Part 3: Reliability and ML-Specific Operations Describe replay, backfills, observability, data-quality controls, and how you prevent an invalid dataset from silently reaching downstream training. #### What This Part Should Cover - Dataset contracts, validation gates, quarantine, and immutable manifests. - Bounded backfills and isolation from normal-priority work. - Metrics for queue delay, task reliability, artifact freshness, lineage gaps, and validation failures. ### What a Strong Answer Covers - A robust distributed scheduler plus ML-data lineage and publication semantics. - Explicit at-least-once execution and idempotent task/output design. - Versioned, reproducible runs and fail-closed validation gates. - Resource fairness, backfills, fault recovery, and actionable operations. ### Follow-up Questions - How would you reuse a task output safely across two workflow runs? - How would you recover if a validation rule itself was wrong? - How would you support a task that expands dynamically into thousands of partitions?