Distributed Job Scheduling And Reconciliation

What's being tested

Interviewers are probing your ability to design and reason about distributed job scheduling and reconciliation mechanisms: scheduling decisions, failure handling, state reconciliation, and operational tradeoffs. They want to see clear assumptions, correctness under partial failure, and pragmatic choices for latency, throughput, and consistency. Expect to justify lease/coordination, idempotency strategies, and how reconciliation (audit/repair) closes gaps left by opportunistic scheduling.

Core knowledge

Scheduler models: difference between centralized (single coordinator) and decentralized (sharded/peer) schedulers; centralized is simpler but single point of failure, decentralized trades complexity for horizontal scale.
Leader election & cluster coordination: use Zookeeper, etcd or Consul for stable leaders and consistent leases; leader churn must be bounded to avoid duplicate scheduling windows.
Leases vs locks: a lease gives temporary ownership; ensure leaseDuration > maxClockSkew + maxTaskExecutionTime to avoid split-brain. Formula: leaseDuration ≥ skew + maxExecution + safetyMargin.
Delivery semantics: at-least-once is easiest (retries), at-most-once requires careful dedup, exactly-once typically implemented via idempotent handlers plus transactional writes (e.g., two-phase commit or idempotency keys).
Idempotency patterns: use persistent idempotency keys (dedup table with TTL), versioned outputs, and safe retries; prefer idempotent ops over distributed transactions for scale.
Task reconciliation (anti-entropy): periodic scans reconcile scheduler state with observed outputs; reconcile window should be longer than expected task variance and backoff to avoid thundering herd.
Failure detection & heartbeats: worker heartbeat interval should balance detection latency vs network load; detect failure if missed N heartbeats, with exponential backoff for reassignments.
Checkpointing & state: for long-running tasks use checkpointed progress in durable store (e.g., Postgres, durable key-value) so reassigned tasks resume rather than restart.
Exactly-once at scale tradeoff: full transactional exactly-once across systems is expensive; prefer idempotent handlers + atomic commit to output store (write-ahead log, tombstones) for practical guarantees.
Throughput & partitioning: shard scheduling by key (user, customer) to reduce contention; maintain global fairness with token-bucket or weighted queues; target N shards so each shard handles ~R/N tasks/sec.
Clock & ordering issues: avoid relying on synchronized clocks; use logical timestamps (monotonic counters) or use clock skew margins. For ordering rely on sequence numbers where needed.
Metrics & SLAs: track reconciliation lag, missedTasks/sec, duplicateExecutions/sec, and p99 reconciliation time; define SLA for how long a missed task must be repaired.

Worked example

Design prompt: "Design a distributed job scheduler that assigns tasks to workers and reconciles missed or duplicate executions." First 30 seconds: clarify task size distribution, guarantees required (at-least-once vs exactly-once), expected throughput, and whether tasks are idempotent. A strong answer organizes around three pillars: (1) assignment (centralized queue vs sharded queues), (2) safety (leases, heartbeats, idempotency), and (3) reconciliation (periodic anti-entropy scans and repair workflows). Choose a pragmatic option: sharded queues keyed by customer, with a small centralized coordinator for metadata. Use leases with leaseDuration = skew + maxExecution + margin and require workers to renew via heartbeat. For reconciliation implement a periodic reconciliation job that compares scheduled/task logs with completed outputs, re-enqueues missing tasks, and marks duplicates using idempotency keys. Flag tradeoff: a short lease reduces latency for failover but increases risk of duplicate execution under clock skew; justify chosen leaseDuration with observed p99 execution time. Close by saying, "If I had more time I'd prototype the idempotency schema and simulate failure scenarios (leader crash, network partition) to tune lease and heartbeat parameters."

A second angle

Alternate prompt: "Implement a reconciliation-only service that repairs missed outputs for a streaming pipeline." Here the scheduler is external; focus shifts to detecting divergence between source (Kafka) offsets and downstream Postgres writes. Use watermarking to determine safe-to-reconcile windows, and build an anti-entropy job that queries offsets and compares counts/hashes per partition. Where reprocessing is required, use idempotent bulk consumers and maintain a reconciliation cursor so the job is resumable. The same concepts apply—durable progress, idempotency keys, and bounded windows—but emphasis changes: detect via aggregations/hashes and repair via safe replays rather than immediate rescheduling.

Common pitfalls

Pitfall: Assuming synchronous exactly-once across heterogeneous services without accounting for latency and partial failures — this leads to blocking or complicated distributed transactions. Prefer idempotency + atomic commit where possible.

Pitfall: Choosing very short lease and heartbeat intervals to "fail fast" without measuring network noise — this increases false failovers and duplicates; benchmark in realistic network conditions.

Pitfall: Treating reconciliation as rare; neglecting operational cost. If you rely on a reconciliation job, plan throttling, backoff, and observability; otherwise the repair system can overload the same services it’s trying to fix.

Connections

These topics frequently pivot to stream processing (checkpointing, watermarks), distributed consensus (Paxos/Raft), and task orchestration platforms (Kubernetes controllers, Airflow) depending on scale and operational model. Be ready to discuss how your scheduler integrates with monitoring, alerting, and deployment tooling.