Design a Distributed Scheduler with At-Most-Once Runs

Design a distributed background-job scheduler with one-time and recurring jobs, an execution SLA, logs, cancellation, and subscriber notifications. The central requirement is **at-most-once execution per scheduled run**. ### Constraints & Assumptions Keep that requirement explicit when discussing crashes and retries. A run that may have started cannot simply be re-executed under a new lease while still claiming strict at-most-once execution. Explain the availability and completion tradeoff, and clarify whether the product actually means at-most-once execution or deduplicated effects. ### Clarifying Questions Does “execution” mean starting application code or committing an external effect? Is a lost or unknown run acceptable? Can a retry be a new scheduled run? Does the SLA measure start or completion? Must subscribers recover missed notifications? ### What a Strong Answer Covers APIs for create/status/cancel/history, job and execution identities, durable claims, queue failure behavior, deadline reporting, logs, and a notification outbox. Justify a Redis queue or any alternative instead of assuming it is sufficient for durability. ### Follow-up Questions What happens if a worker dies immediately before or after beginning execution? What if the queue loses messages or the database commit times out? How do you push or broadcast updates to subscribers without losing durable status history?