Design Scheduler Claims, Execution Guarantees, and Recovery

Design the reliability mechanisms for a distributed job scheduler with multiple executors. Address three questions: how to ensure a scheduled work item runs only once, how to prevent several executors from taking the same work, and how to handle errors and fallbacks. Distinguish the guarantee a scheduler can provide about a claim or attempt from the guarantee it can provide about the job's external effects. Explain what additional cooperation the job handler or destination system would need. ### Constraints & Assumptions - Multiple executors may compete for eligible work. - Storage technology, delivery transport, job side effects, and workload scale are unspecified; state your choices. - Treat a process stopping between operations or an acknowledgement being lost as failure scenarios to analyze, not as proof that the job did nothing. - Do not assert exactly-once effects solely because a lock or queue exists. ### Clarifying Questions to Ask - Does "only once" mean one accepted schedule, one active attempt, one completed handler invocation, or one committed business effect? - Can the work and its completion record participate in the same transaction? - Are job handlers idempotent, and can external destinations accept an idempotency key or fencing token? - Which errors can be retried automatically, and which require a permanent failure or operator decision? ### Part 1 — Claim Work Safely Propose a durable work record and a claim protocol for competing executors. Explain how a claim expires or is relinquished and how an old executor is prevented from corrupting a newer attempt's result. #### What This Part Should Cover - An atomic ownership decision, rather than a read followed by an unconditional update. - Work identity, attempt identity, and an explicit lifecycle. - Recovery after an owner stops, including the possibility that an expired owner later resumes. - The scope and limitations of leases or fencing. ### Part 2 — Define the Execution Guarantee Walk through failures immediately before the job's effect, immediately after the effect, and before the completion acknowledgement. Explain which duplicate or lost-work risks remain and how the design handles them. #### What This Part Should Cover - A separation between retryable attempts and deduplicated business effects. - A valid mechanism for atomic effect-and-completion when that is possible. - Idempotency or reconciliation when an effect occurs in an external system. - An honest description of guarantees when the destination offers neither transactions nor deduplication. ### Part 3 — Handle Errors and Fallbacks Describe retry policy, terminal failure handling, and any fallback behavior. Explain how you would observe failures without allowing recovery logic to execute the same business effect through an uncoordinated second path. #### What This Part Should Cover - Classification of transient, permanent, and indeterminate failures. - Bounded retries, delayed reattempts, and useful failure records. - The semantics of any fallback and its relationship to the original work identity. - Evidence that distinguishes work waiting normally from abandoned or repeatedly failing work. ```hint A worker whose lease expired may still be alive. Trace what happens if it resumes after a replacement worker has begun. ``` ### What a Strong Answer Covers A complete design defines exactly what is protected by the claim protocol and what is protected by the effect protocol. It walks through ambiguous outcomes, explains the conditions required for stronger guarantees, and makes error handling preserve those same invariants. ### Follow-up Questions - What if the destination processes a request but the caller times out before receiving its response? - Can a distributed lock alone guarantee exactly-once email delivery? - How would an operator safely retry a job marked as having an indeterminate outcome?