# Design a Task Scheduler and Executor Design a service that accepts tasks for future or recurring execution and dispatches them to workers. ### Constraints & Assumptions - Practice assumption: tasks include one-time, fixed-delay, and cron-like schedules. - Cron and fixed-rate schedules advance from their intended occurrence when dispatch is durably created. A fixed-delay schedule creates its next occurrence only after the current occurrence reaches its configured terminal completion state; retries remain part of the same occurrence. - At peak, clients create 100000 tasks per second and 100 million tasks may be pending. - Execution is at least once; clients can supply an idempotency key. - A task may run for seconds or hours and can be retried. - Dispatch within one second of the due time is acceptable under healthy conditions. ### Clarifying Questions to Ask - Is exactly-once business effect required, or only exactly-once scheduling intent? - How are task payloads stored, secured, and versioned? - What retry, cancellation, priority, and tenant-isolation semantics are required? - Can recurring executions overlap? - For fixed-delay work, does the delay begin after success, any terminal result, or final retry exhaustion? - How should cron time zones, daylight-saving transitions, and missed occurrences behave? ### Part 1: API and Data Model Define creation, update, cancellation, inspection, and execution-attempt records. Explain state transitions and concurrency control. #### Hints - A logical task and one attempt are not the same entity. #### What This Part Should Cover - Stable identifiers and idempotency - Explicit states and versions - Schedule representation - Distinct advancement rules for cron, fixed-rate, and fixed-delay schedules ### Part 2: Finding and Dispatching Due Work Explain how schedulers find due tasks without scanning the full table, avoid duplicate ownership, and feed executors with backpressure. #### Hints - Consider partitioning by both time and ownership. #### What This Part Should Cover - Due-time indexing - Safe concurrent claiming - Queueing and overload behavior ### Part 3: Execution, Retry, and Recovery Describe worker leases, heartbeats, retry policy, poison tasks, cancellation races, and recovery after scheduler or worker failure. #### Hints - Separate acknowledgment of dispatch from completion of business work. #### What This Part Should Cover - At-least-once consequences - Bounded retry and dead-letter handling - Idempotent creation of the next fixed-delay occurrence after terminal completion - Observability and reconciliation ### What a Strong Answer Covers - A consistent task and attempt state machine - A scalable due-work index and partitioning strategy - Fenced or transactional claims with backpressure - Honest delivery semantics, idempotency, recovery, and tenant isolation - Correct recurrence semantics when runs are long, retried, or completed more than once at the protocol boundary ### Follow-up Questions - How would you support tasks scheduled years in the future? - What happens when a tenant schedules a million tasks for the same second? - How would you change the design for sub-100-millisecond precision? - How do you deploy a worker code change while old tasks remain queued?