Design a job scheduler with SLA and logs

Scenario

Design a distributed job scheduler that runs scheduled (time-based) jobs only (e.g., cron/interval). Each job execution has an SLA: if the run does not finish within the SLA window, the system must mark it as timed out and raise an error/alert. Users must be able to inspect execution logs.

Core requirements

• Create/update/delete scheduled jobs.
• Trigger job runs at the correct times (support cron-like schedules or fixed intervals).
• Execute jobs on a fleet of workers.
• Track job run state: PENDING/RUNNING/SUCCEEDED/FAILED/TIMED_OUT/CANCELED .
• SLA enforcement per run: detect overruns and surface as an error/alert.
• Provide log viewing per job run.

Non-functional requirements (clarify in discussion)

• Scale: potentially many jobs and high execution rate.
• Reliability: tolerate worker crashes, scheduler crashes, network partitions.
• Correctness: avoid missed schedules; minimize duplicate runs (define acceptable semantics).
• Observability: metrics, tracing, auditing.

Follow-ups to address

• How to handle failures at each stage (scheduler failure, worker failure, log pipeline failure, DB outages).
• Retries/backoff, dead-lettering, and idempotency.
• Handling long-running jobs and cancellation.
• Handling clock skew/time zones and daylight savings.