# Design a CI/CD Job Scheduler Design the job scheduler behind a CI/CD platform. It accepts pipeline definitions, expands dependency graphs into runnable jobs, places work on heterogeneous workers, streams status, retries failures, and supports cancellation. Cover fairness, resource matching, artifacts, secrets, idempotency, worker loss, and high availability. ### Constraints & Assumptions - A pipeline is a directed acyclic graph of jobs with declared CPU, memory, platform, and optional accelerator needs. - Thousands of tenants may submit bursty workloads. - Workers can disconnect or die while a job is running. - Some jobs are safe to retry; deployments may require explicit idempotency controls. - Logs and artifacts can be much larger than scheduler metadata. ### Clarifying Questions to Ask - What submission rate, concurrent-job count, duration distribution, and startup target apply? - Which operating systems, isolation levels, and hardware pools are required? - Are priorities, quotas, reservations, or preemption supported? - What ordering and exactly-once expectations exist for deployment jobs? ### What a Strong Answer Covers - Durable pipeline and job state with dependency-aware readiness transitions - A scalable queueing and placement design with quotas, fairness, and resource matching - Worker leases, heartbeats, fencing, retries, cancellation, and reconciliation - Separate log, artifact, cache, and secret paths - Scheduler high availability, backpressure, security, and operational metrics ### Follow-up Questions 1. How would you prevent a recovered worker from reporting success after its lease was reassigned? 2. How would you schedule a scarce GPU pool fairly across tenants? 3. Which deployment steps should never be retried automatically? ```hint Make every attempt uniquely identifiable Lease and fence execution attempts so a late result from a worker that lost ownership cannot overwrite the current job state. ```