Design a service that accepts a list of parameters, uses an external asynchronous cluster API submit_job(params) -> job_uuid and check_job_status(job_uuid) -> ['SUBMITTED','RUNNING','SUCCEEDED','FAILED'], executes the jobs, and notifies the user when all jobs in the list have completed. Specify the architecture and components (ingest API, scheduler, worker pool, status tracker, persistent store, notification service). Detail the execution flow for fan-out submission, concurrency and rate limiting, retry/backoff and idempotency for submissions and status checks, persistence for crash recovery, handling of partial failures and retries for FAILED jobs, timeouts and detection of stuck jobs, deduplication, and cancellation. Explain how you would scale to millions of jobs and choose between polling versus event-driven callbacks if available. Define client- and server-side APIs, notification semantics (exactly-once vs at-least-once), and monitoring/alerting and observability.

Design async job orchestration and notification service evaluates requirements, scale assumptions, API/data design, architecture, trade-offs, failure modes, and rollout in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

How do I approach System Design interview questions?

System Design questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master system design interviews.

What difficulty level is this interview question?

This is a hard difficulty System Design question, commonly asked during Technical Screen rounds at Amazon.

What role is this question designed for?

This question is commonly asked for Machine Learning Engineer candidates at Amazon during technical interviews.

Design async job orchestration and notification service

System Design: Batch Orchestration Over an External Asynchronous Cluster

Context

You are designing a backend service that accepts a list of job parameters from clients, submits each job to an external asynchronous compute cluster, and notifies the client when all jobs complete. The external cluster exposes two APIs:

submit_job(params) -> job_uuid
check_job_status(job_uuid) -> one of ['SUBMITTED', 'RUNNING', 'SUCCEEDED', 'FAILED']

Assume the external API is eventually consistent, has rate limits, may transiently fail, and does not guarantee idempotency unless you add your own keys. The cluster may optionally support callbacks/webhooks on job completion; otherwise, you must poll.

Requirements

Design the service with the following components and details:

Architecture and components

Ingest API
Scheduler
Worker pool (submitters and status pollers)
Status tracker and aggregator
Persistent store
Notification service

Execution flow

Fan-out submission from a list of parameters
Concurrency control and rate limiting (per-tenant and global)
Retry, backoff with jitter, and idempotency for both submissions and status checks
Persistence for crash recovery and exactly-once-orchestrator semantics
Handling partial failures, including automatic retries for FAILED jobs based on policy
Timeouts and detection of stuck jobs (SUBMITTED/RUNNING too long)
Deduplication (same input re-sent by client or internally retried)
Cancellation (single job or entire batch)

Scale and strategy

How to scale to millions of jobs and high QPS
Polling vs event-driven callbacks (if available): trade-offs and hybrid approach

APIs and semantics

Define client-facing APIs and server-to-server APIs if applicable
Notification semantics (exactly-once vs at-least-once); integrity and dedup

Operations

Monitoring, alerting, and observability (metrics, logs, traces; SLOs)

Keep the design practical and production-ready, with clear assumptions and trade-offs.

Constraints & Assumptions

Preserve the scope, facts, inputs, and requested outputs from the prompt above.
If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.

Clarifying Questions to Ask Guidance

Clarify users, core use cases, read/write patterns, scale, latency, availability, and data retention.
State explicit assumptions before making sizing or architecture decisions.
Prioritize the functional path first, then address reliability, security, observability, and rollout.

What a Strong Answer Covers Guidance

A scoped requirements summary with concrete non-goals and success metrics.
API, data model, architecture, consistency, capacity, and operations.
Reasoned trade-offs among simple and scalable designs, including bottlenecks and failure modes.
A validation, monitoring, migration, and launch plan appropriate for the risk level.

Follow-up Questions Guidance

What breaks first at 10x traffic or data volume?
How would you degrade gracefully during dependency failures?
What metrics and alerts would prove the design is healthy after launch?