System Design: Scheduled Payments Service (Take‑home)

Context

You are designing a backend service for a consumer payments platform that supports one‑off, future‑dated payments. The system must maintain strong ledger correctness and provide operational reliability at scale.

Assume there is an existing Ledger/Transfer service that performs actual debits/credits atomically and supports idempotency. You will design the Scheduled Payments service and its scheduler/executor.

Requirements

1. Functionality
   • Create a future‑dated payment (one‑time only).
   • List and get details of scheduled payments.
   • Execute the payment at or after its due time.
   • Cancel a scheduled payment if it has not executed.
2. Interfaces
   • Specify REST and gRPC APIs (endpoints/RPCs, request/response fields, error codes).
   • Idempotency keys for write operations and exact semantics.
   • Authorization model (user vs. internal services).
   • Input validation rules.
3. Architecture
   • Propose a scheduler design (e.g., time‑indexed buckets, distributed delayed queues) that ensures exactly‑once execution from a ledger perspective.
   • Address time zones and daylight savings transitions.
   • Retries with backoff and jitter; classify transient vs. permanent failures.
   • Handle race conditions between execution and cancellation.
   • Deduplication and poison message handling.
   • Monitoring, alerting, and operational dashboards.
   • Backfilling missed jobs after outages.
4. Data and Concurrency
   • Define the data model (tables/entities, key fields, indexes).
   • Concurrency control strategy and state machine; include how you prevent double execution.

Deliverables

• REST/gRPC API specifications with idempotency and auth.
• End‑to‑end architecture of scheduler and executor ensuring exactly‑once ledger effects.
• Policies for time zones/DST, retries, races, dedup, poison messages, monitoring, and backfill.
• Data model and concurrency control plan.
• Clearly state assumptions where necessary.