Evaluation of a Distributed Task Scheduler's Design

Requirements compliance

Evaluate submission, dispatch, and execution separately. A reachable submitter is useful only if it can durably accept work, and a queued task still needs a recoverable execution owner.

Availability

Replicate submitters, queues, and scheduler metadata across the chosen failure domains. Failover must preserve ownership and prevent stale workers from committing conflicting results. Monitor required storage and routing dependencies as well as worker health; distribution alone does not ensure that admission or dispatch remains available.

Durability

A successful submission response follows durable task storage. Keep task lifecycle state after queue publication, and use a reliable handoff so a crash between the database write and queue send cannot lose work. Queue delivery can repeat, so consumers must check the task’s current state and attempt identity.

Scalability

Scale the measured bottleneck: submission throughput, metadata transactions, queue operations, or worker capacity. Partition ownership deliberately and watch for hot tenants or dependency graphs. Adding queues does not create compute capacity, and adding workers does not fix a serialized metadata hot spot.