Design a Cron Scheduler with At-Least-Once Delivery and Independent Monitoring

Design a job scheduler for recurring cron jobs only. The delivery guarantee is at least once. Explain monitoring, log storage, and how transactions and an outbox can support reliable dispatch without creating unnecessary bottlenecks. ### Constraints & Assumptions - Recurring cron schedules are the scope; one-off job submission is not required. - Duplicate delivery is possible and must not be described as exactly-once execution. - Time zones, missed schedules, overlapping executions, and worker capacity need explicit policies. - Monitoring should detect a failed scheduler independently of that scheduler's own reporting path. ### Clarifying Questions to Ask - What happens to missed occurrences after downtime? - Can one schedule have overlapping runs, and which time zone defines its calendar? - Which logs need live inspection versus long-term object storage? ### Part 1 — Create and Dispatch Occurrences Explain recurrence calculation, occurrence identity, transactional state, and outbox delivery. #### What This Part Should Cover - Stable identities that prevent accidental duplicate occurrence creation. - At-least-once dispatch and worker idempotency expectations. - Bounded scans, batching, and retention of dispatch evidence. ### Part 2 — Detect Failure and Retain Logs Describe an independent watchdog and storage/query paths for execution logs. #### What This Part Should Cover - Heartbeat and progress signals observed outside the scheduler. - Per-occurrence status, execution attempts, and retrievable logs. - Alerting for lag and missing completion, not just process availability. ```hint A scheduler cannot reliably notify you through its own loop when that loop has stopped. ``` ### What a Strong Answer Covers The design should connect calendar occurrences to durable delivery and observable completion. Optimizing the outbox must preserve atomic intent recording, and monitoring must distinguish a live process from successful scheduling. ### Follow-up Questions - How would daylight-saving transitions affect a cron expression? - What if the relay publishes an outbox record but crashes before marking it sent? - How would a large backlog avoid starving current schedules?