Design a distributed job scheduler service

Design a **distributed job scheduling system** (microservice-based, running in the cloud) with the following requirements: ### Functional Requirements 1. **Create scheduled jobs** - A client can create a job via an API. - For each job, the client specifies: - Job identifier (optional; otherwise generated by the system). - A **schedule** (e.g., cron-style, or "run at time T", or "every X minutes"). - The **task** to run (e.g., a script name, container image, or some executable description). - Resource requirements (e.g., CPU/memory tier or machine type) at a high level. - A **timeout** value. - A **timeout handler** (what to do if the job exceeds its timeout: kill, mark failed, retry, etc.). 2. **Run jobs reliably at the designated time** - Each job should run as close as possible to its scheduled time. - The system is **distributed**: jobs can run on many worker machines / microservices. - The system must tolerate machine failures and restarts. 3. **Handle jobs that don't finish on time** - If a job exceeds its declared timeout, the system must: - Detect this condition. - Execute the configured timeout behavior (e.g., kill the job, mark as failed, trigger a handler job, or reschedule, depending on your design). 4. **Query past job status** - Provide an API to query: - The status of a specific job run (e.g., PENDING, RUNNING, SUCCESS, FAILED, TIMED_OUT). - The history of runs for a job (e.g., last N runs, with timestamps and outcomes). 5. **Query job logs** - Provide an API to fetch **logs** for a past job run (e.g., stdout/stderr or structured logs). 6. **Distributed microservice model** - Assume the interviewer wants a **microservice-based**, horizontally scalable architecture. - Components should be cloud-hosted services (e.g., on containers/VMs) and able to scale independently. ### Non-Functional Requirements (assume reasonable scale) - Support on the order of **hundreds of thousands** of active scheduled jobs. - Jobs may be scheduled with **minute-level granularity** (or better, if you choose). - System should be **fault tolerant**: if a scheduler instance or worker crashes, jobs should still eventually run. - Favor **reliability over perfect exact timing** (e.g., running a job a few seconds late is acceptable, but skipping it completely is not). ### Expected Discussion Describe and justify: 1. **High-level architecture** - What microservices you would define (e.g., API service, job metadata service, scheduler, worker/runner service, log service, etc.). - How these services communicate (HTTP/REST, message queues, etc.). 2. **Data model and storage** - How you store job definitions and schedules. - How you store job run history and statuses. - How and where you store logs (e.g., blob/object storage vs database vs log store). - Choice of databases (relational vs NoSQL) and why. 3. **Scheduling logic** - How the system determines **which jobs to run at what time**. - How you implement a **distributed, fault-tolerant scheduler**: - How many scheduler instances are running. - How they share work or elect a leader. - How you avoid double-scheduling or missing jobs. 4. **Job execution** - How jobs are dispatched to worker machines. - How you track job start/end, collect exit codes, and enforce **timeouts**. - How you communicate logs back to the system. 5. **Reliability and failure handling** - What happens if a scheduler instance crashes. - What happens if a worker dies mid-job. - How you avoid running the **same job twice** vs. accepting at-least-once execution with idempotent jobs. - How you recover after a restart (e.g., replay job state from durable storage). 6. **APIs for querying status and logs** - Example REST endpoints for: - Creating/updating/deleting jobs. - Fetching job configuration. - Fetching job run history. - Fetching logs for a specific run. 7. **Scalability considerations** - How you scale horizontally when the number of jobs or workers increases. - Any sharding or partitioning of job data or schedules. Explain your design step by step, call out trade-offs, and explicitly mention any assumptions you make (e.g., exact vs approximate scheduling accuracy, at-least-once vs exactly-once execution semantics, log retention policies, etc.).