Distributed Job Scheduling For Financial Pipelines
Asked of: Software Engineer
Last updated
What's being tested
This topic tests whether you can design a distributed scheduler that executes dependent financial jobs reliably despite crashes, retries, duplicate messages, delayed workers, and changing workloads. The interviewer is probing your command of state machines, failure semantics, concurrency control, dependency scheduling, and operational safety—not merely whether you can enqueue tasks. Ramp cares because financial workflows require predictable completion, strong auditability, tenant isolation, and correctness when an operation is retried or partially completed.
Core knowledge
-
A scheduler should model each job as a durable state machine:
`PENDING` → `READY` → `RUNNING` → `SUCCEEDED`, with explicit`FAILED`,`RETRY_WAIT`,`CANCELLED`, and possibly`PAUSED`states. Persist transitions so a restarted scheduler can reconstruct truth. -
Represent dependencies as a DAG rather than recursively scanning all jobs. A task becomes runnable when every predecessor succeeds; maintain a remaining-dependency count and decrement it transactionally as upstream tasks complete.
-
Separate job intent from execution attempt. A logical job may have many attempts, each with an attempt ID, worker lease, start time, finish time, error, and retry number. This preserves audit history and makes retry behavior explainable.
-
Use a durable queue such as
`Kafka`,`SQS`, or a database-backed work table, but do not treat delivery as completion. Workers should acknowledge only after committing their result, and the scheduler must reconcile queued, running, and completed records. -
Distributed workers need leases with expiration timestamps and heartbeats. A worker that stops heartbeating becomes eligible for reassignment, but lease expiry can create concurrent execution; use fencing tokens or attempt generations so stale workers cannot commit afterward.
-
At-least-once delivery is usually the practical baseline. Make handlers idempotent using a stable operation key such as
`(workflow_id, logical_job_id, input_version)`, a uniqueness constraint, and transactional result recording; “exactly once” is generally an end-to-end illusion. -
Retry policy must distinguish transient from permanent failures. Use bounded exponential backoff with jitter, for example , plus a maximum attempt count and a dead-letter or manual-review state.
-
Control concurrency at multiple levels: global worker capacity, per-tenant quotas, per-workflow limits, and resource-specific locks. Fair scheduling such as weighted fair queuing prevents one large customer or backfill from starving latency-sensitive work.
-
Preserve ordering only where required. Partitioning by account, entity, or workflow can provide ordered processing, but reduces parallelism; explicitly identify the smallest ordering key that protects financial invariants.
-
Handle backpressure with bounded queues, admission control, and overload signals. Monitor queue age and runnable-job count, not only throughput; a useful freshness measure is
`now - scheduled_time`at execution start, reported by tenant and priority. -
Make scheduling decisions transactionally safe. A
`Postgres`row lock, compare-and-swap version, or atomic claim query can prevent two schedulers from claiming the same job; avoid holding database transactions open while remote work executes. -
Design for reconciliation and auditability. Store immutable transition events, actor or service identity, correlation IDs, input/output versions, and reason codes; a periodic reconciler should detect jobs stuck in
`RUNNING`, missing completions, or violated dependency counts.
Worked example
Representative prompt: “Design a Distributed Job Scheduler for Financial Pipelines.” In the first 30 seconds, clarify whether schedules are one-time or recurring, the expected job volume and latency, dependency shapes, retry tolerance, tenant isolation, and whether external side effects are reversible. Declare an assumption such as at-least-once execution with idempotent job handlers, durable state in `Postgres`, and a queue for dispatch. Organize the answer around four pillars: the workflow and execution state model, dependency-aware dispatch, lease-based worker failure recovery, and observability/reconciliation. Explain that a scheduler transaction claims a `READY` job, assigns an attempt and fencing token, and publishes work, while workers persist success or failure and heartbeat during execution. Explicitly flag the claim-versus-publish tradeoff: committing the database claim before publishing risks lost work, while publishing first risks duplicates, so use an outbox pattern or a transactional queue integration. Mention that external financial effects require an idempotency key because scheduler-level deduplication cannot prevent a timeout after the remote system accepted a request. Close by saying that with more time you would quantify capacity, define tenant fairness, and walk through a scheduler crash at every transition.
A second angle
Representative prompt: “How should the scheduler recover when workers, queues, or dependencies fail?” The framing shifts from the normal architecture to failure timelines and invariants: no permanently lost work, no stale worker committing after reassignment, and no downstream job running before required inputs are durable. Discuss lease expiry, fencing tokens, retry classification, dead-letter handling, and reconciliation scans. Also distinguish a failed prerequisite from a temporarily unavailable prerequisite, since those should produce different downstream behavior and operator actions.
Common pitfalls
Pitfall: Saying “exactly-once processing” without explaining the external side effect boundary is a common analytical mistake. A stronger answer states that delivery is at least once and uses idempotency keys, durable deduplication, and reconciliation to make the business operation effectively once.
Pitfall: Jumping directly to
`Kafka`versus`SQS`misses the design problem. First define job states, ownership, dependency readiness, retry semantics, and correctness invariants; then select infrastructure that supports those requirements.
Pitfall: Treating a worker timeout as proof that the job did not run is a depth mistake. The worker may have completed the financial operation before losing connectivity, so recovery must reconcile with the downstream system or safely repeat an idempotent request.
Connections
An interviewer may pivot to the outbox pattern, sagas, leader election, rate limiting, or event sourcing. Be prepared to compare a custom scheduler with workflow engines such as `Temporal`, `Airflow`, or `Cadence`, especially around durability, long-running timers, retries, and operational ownership.
Further reading
-
Designing Data-Intensive Applications by Martin Kleppmann — strong treatment of replication, stream processing, consistency, and distributed-system failure modes.
-
Release It! by Michael T. Nygard — practical guidance on timeouts, bulkheads, circuit breakers, and production failure handling.
-
Temporal Documentation: Durable Execution — concrete examples of durable workflows, activity retries, timers, and recovery after process failure.