Design a Delayed Job Scheduler (LLD)

Design a service that schedules a job to execute X seconds in the future with second-level accuracy. Produce a low-level design that covers APIs, data structures, persistence, execution model, and operational concerns.

Assume:

• Second-level precision is sufficient.
• The system must survive process restarts without losing scheduled jobs.
• At-least-once delivery is acceptable by default; discuss options for exactly-once.

Requirements

1. APIs
   • schedule(job, delaySeconds) -> jobId
   • scheduleAt(job, epochMillis) -> jobId
   • cancel(jobId) -> success/failure
   • getStatus(jobId) -> job metadata
   • Optional: reschedule(jobId, newDelaySeconds)
2. Data structures: propose and justify (e.g., min-heap by due time, timing wheel, or hybrid) including time/space complexity.
3. Persistence and durability: how jobs are durably stored and recovered after restarts/crashes.
4. Worker execution model: how jobs are dispatched and executed; leasing/ack model.
5. Restarts and clock drift: recovery logic and timekeeping choices.
6. Delivery semantics: idempotency, at-least-once vs exactly-once trade-offs.
7. Concurrency limits: global and per-tenant limits; fairness.
8. Time complexity: for schedule, cancel, due job retrieval.
9. Class diagrams: main classes, relationships, and key methods.
10. Testing: strategy across unit, integration, reliability, and performance.