System Design Prompt: CI/CD Platform (shell-script jobs)
Design a CI/CD system that can:
-
Allow users to define
pipelines
consisting of multiple
jobs
, where each job is just a
shell script
.
-
Trigger pipeline runs via
git events
(push/PR) and also manually.
-
Schedule jobs onto a fleet of workers, respecting job dependencies (DAG) and basic resource constraints (e.g., concurrency limits).
-
Persist run history: pipeline status, job status, timestamps, exit codes.
-
Store and retrieve job logs.
-
Stream job output logs in near real-time
to a “status service”/UI while the job is running.
Out of scope
-
Do not assume containers/images (no container registry, no Docker/K8s specifics).
What to cover
-
High-level architecture and core services.
-
Data model for pipelines/runs/jobs.
-
Execution model (how scripts run on workers), retries, idempotency.
-
Log ingestion, storage, and real-time streaming to users.
-
Scalability, reliability, and security considerations (secrets, isolation).