Design a Distributed System for Parallel Job Execution

Context

You are asked to design a highly available, horizontally scalable service that executes many independent tasks (embarrassingly parallel jobs) across a compute cluster. Assume multi-tenant usage, heterogeneous task durations (milliseconds to hours), and the need for strong operational visibility and robust fault tolerance.

Requirements

Specify the following, with clear APIs, data models, and rationale:

1. Job model and external APIs
2. Task partitioning and sharding strategy
3. Scheduler and worker architecture
4. Coordination and concurrency control (leases, heartbeats, visibility timeouts)
5. Fault tolerance, retries, backoff, and dead-letter handling
6. Idempotency and deduplication approach
7. Progress tracking and result aggregation
8. Scaling and resource management (autoscaling, quotas, bin-packing)
9. Backpressure and prioritization
10. Ordering guarantees and trade-offs
11. Consistency semantics (at-least-once vs exactly-once) and trade-offs
12. Monitoring, alerting, and observability
13. Capacity planning assumptions and calculations

State any minimal assumptions you need to make the design concrete.