Design an At-Most-Once Job Scheduler

Design a distributed job scheduler that supports both one-time and recurring jobs. Requirements: - Clients can create jobs with a schedule, payload, retry policy, and notification target. - Each scheduled run should be owned by at most one worker at a time. - Duplicate successful execution of the same scheduled run should be avoided. - The system should support retries for transient failures. - For recurring jobs, a failed run must still generate a failure notification for that specific occurrence. - If a scheduler, worker, or notification component crashes, the system should recover without silently losing jobs or notifications. Discuss the API, storage model, scheduling logic, worker coordination, retries, notification delivery, and how you would handle component failures and recovery.