## Scenario Design a scheduler for payment jobs. A client can create a payment instruction to execute once at a future time, inspect its state, and cancel it before execution begins. The scheduler must tolerate duplicate requests, worker crashes, temporary provider failures, and delayed execution without charging twice. Cover API semantics, storage, due-job discovery, work claiming, idempotency, retries, cancellation races, reconciliation, scaling, and monitoring. Do not assume that an external payment provider participates in your database transaction. ### Constraints & Assumptions - A payment instruction has a caller-supplied idempotency key, amount, currency, destination reference, and execution time. - Exactly-once side effects cannot be guaranteed by message delivery alone. - The system must preserve an immutable history of state transitions. - A job may execute late during an outage, but never before its scheduled time. - Cancellation is allowed until the atomic transition that marks provider submission as started; define that transition and every pre-submission state precisely. ### Clarifying Questions to Ask - What lateness target and job volume must the scheduler support? - Can a client cancel after a provider request has started? - If cancellation races a worker claim or submission-start transition, which compare-and-swap wins and what exact API result follows? - Does the provider accept an idempotency key and expose status lookup? - Which failures are retryable, and when is human review required? - Are recurring payments in scope? No for the baseline. ```hint Make claiming recoverable Persist a lease with owner and expiry, then let another worker reclaim it after a crash. A lease is not the payment idempotency mechanism. ``` ### What a Strong Answer Covers - A durable payment state machine and uniqueness on the client idempotency key. - Indexed time buckets or partitions for due-job discovery without full scans. - Transactional work publication, leases, retry scheduling, and provider idempotency. - Clear cancellation-versus-execution linearization, CAS transitions from every pre-submission state, deterministic race responses, and a reconciliation loop for unknown outcomes. - Auditability, encryption and access control, rate limits, and business-level invariants. ### Follow-up Questions 1. What state should represent a provider timeout after the request may have succeeded? 2. How do you prevent a hot execution-time partition at the top of an hour? 3. How would recurring schedules handle daylight-saving transitions?