This question evaluates a candidate's competency in distributed systems and messaging architecture, including scalability, high-throughput design, large-payload storage strategies, API/resource modeling, reliability and multi-tenant operational concerns.
Design a distributed message queue system that supports high-throughput producers and consumers with large payloads. Start from APIs for topics or queues (create, update, delete, list), produce and consume semantics, and deployment management. Explain strategies to handle failures, prevent data loss, and recover (acks, retries, deduplication). Contrast an in-memory queue class and API with a multi-tenant cloud service; describe storage choices, partitioning, replication, and resource isolation among different users.
Quick Answer: This question evaluates a candidate's competency in distributed systems and messaging architecture, including scalability, high-throughput design, large-payload storage strategies, API/resource modeling, reliability and multi-tenant operational concerns.
Design a distributed message queue system that supports very high throughput for both producers and consumers and can handle large message payloads.
Your design must address four areas:
APIs and resource model
— CRUD on topics/queues; produce and consume semantics including batching, acknowledgments, consumer groups, ordering, and offset management; and deployment/administration (provisioning, scaling, health, metrics, upgrades).
Reliability and recovery
— strategies to prevent data loss and handle failures: acknowledgments, retries with backoff, dead-letter queues (DLQ), deduplication, and recovery from broker or consumer failures.
Architecture and trade-offs
— contrast a single-process, in-memory queue
class + API
against a multi-tenant, cloud-hosted
service
. Cover storage choices for large payloads, the partitioning/sharding strategy, replication and replica placement, and resource isolation across tenants.
Performance
— throughput, latency, batching, flow control/backpressure, and scaling strategies.
Provide a clear design, concrete API sketches, and the rationale behind your key choices.
Constraints & Assumptions
State your assumptions explicitly; reasonable defaults to work with:
Throughput target:
size for ~1 GB/s of useful (post-compression) ingress at the cloud service. Account for both a small-message profile (~1 KB avg) and a large-message profile (~1 MB avg, up to ~100 MB) because they stress the system very differently (message-rate-bound vs. byte/IO-bound).
Durability:
no acknowledged data loss; survive the loss of a single availability zone.
Availability:
high availability for the data plane via leader failover.
Latency:
low-tens-of-ms p99 for produce-ack at quorum durability, tunable down by relaxing acks.
Ordering contract:
total order
within a partition only
— global total ordering is out of scope.
Multi-tenancy:
many tenants share the fleet; one tenant must not be able to starve another.
Out of scope:
strong cross-partition/cross-topic transactions, global total ordering, and a SQL query layer over messages.
Clarifying Questions to Ask
What delivery semantics does the workload need — at-least-once, at-most-once, or effectively-once — and is that a global choice or per-topic?
What is the message-size distribution and the maximum payload? (Drives the inline-vs-claim-check threshold.)
What ordering guarantee is required — per-key, per-partition, or global? (Global total order conflicts with horizontal scale.)
What retention is needed (time or bytes), and how long must consumers be able to replay?
How many tenants/topics/partitions, and how skewed are the partition keys? (Hot-partition risk.)
Is this a streaming workload (offset/commit) or a task-queue workload (per-message ack + DLQ), or both?
What a Strong Answer Covers
Storage primitive:
an append-only partitioned log with immutable segments + sparse index; clear rationale for why a log (not a heap/DB) gives ordering, replay, and write throughput.
Control/data-plane split:
a Raft-backed metadata store for topic config, placement, ISR, and leadership; stateless gateways; a broker fleet owning partition replicas.
API surface:
concrete topic CRUD, produce (with batching/idempotence/acks), and
both
consume contracts (offset/commit and visibility-timeout ack/nack/extend), plus admin/deployment endpoints.
Reliability:
delivery-semantics knob, idempotent producers + dedup window, retries/backoff, DLQ for poison messages, and concrete recovery flows for broker crash, consumer crash, and AZ loss.
Large payloads:
inline-vs-claim-check threshold, blob lifecycle/consistency, and how this bounds broker memory and replication bandwidth.
Partitioning, replication, placement:
key→partition routing, hot-partition mitigation, RF/ISR, leader election, AZ/rack-aware placement, and the increase-only partition-count caveat.
Performance levers:
batching, zero-copy reads, compression, sequential I/O, and credit-based backpressure; scaling via partitions/brokers.
The in-memory vs. cloud contrast:
the near-identical API contract and what the cloud service
adds
(durability, isolation, operations).
Sizing sanity check:
back-of-envelope for broker count, replication amplification, and storage/day that motivates tiered storage.
Follow-up Questions
A single key is hot and saturating one partition. How do you detect it, and what are your options given that increasing partition count re-routes existing keys under
hash(key) % P
?
A consumer needs
effectively-once
end to end (produce → process → produce). Walk through how idempotent producers + transactions achieve it, and where true exactly-once still requires an idempotent sink or outbox/inbox.
You computed ~86 TB/day of pre-replication data with multi-day retention. How does tiered storage change the cost model, and what is the latency penalty for reads served from object storage?
During a rolling upgrade you must restart every broker. How do you do this without dropping availability or losing acknowledged writes?