Metrics Ingestion System Design Interview: Cardinality, Aggregation, and Backpressure

A metrics ingestion system design interview tests whether you can preserve meaning while reducing enormous write volume. The difficult questions appear before storage: what makes two samples the same series, can they be combined, and what should happen when a tenant sends a user ID as a label? If those rules are vague, no amount of Kafka partitions or columnar storage will make the result trustworthy.

The OpenTelemetry specification officially models a metric stream using identifying properties and attributes, and it distinguishes delta from cumulative temporality. Its SDK specification also defines cardinality limits and an overflow attribute set. Prometheus Remote Write documents an HTTP protocol for samples and operational retry buffering. Those sources define behavior of interfaces and SDKs; this article's capacity numbers and architecture are interview inferences.

Use Design a Metrics Collection System as the closest PracHub drill while this guide develops the broader interview decision. PracHub question records are candidate-reported practice material: each record supports its own prompt, not a claim that every employer asks the same question.

Metrics Ingestion System Design Interview: Quick Answer

Define a series key as (tenant, metric name, resource identity, attribute set, aggregation semantics). Normalize it once, hash it for routing, and keep tenant identity outside user-controlled labels. Receive bounded batches, authenticate and validate them, append accepted data to a durable write-ahead log or broker, then aggregate and write query-optimized blocks asynchronously. Acknowledgment semantics must say whether 200 means “validated,” “durably queued,” or “stored in the final database.”

Bound active-series cardinality per tenant and per metric. Reject, drop selected labels, or route overflow through a documented aggregate; never merge arbitrary series invisibly. When storage slows, allow durable queue lag to grow within retention, reduce optional work, and apply explicit per-tenant admission control. Expose oldest-unprocessed age, queue bytes, rejection count, active series, compression ratio, and end-to-end sample availability.

Start with the Query and Series Contract

Ask whether users need raw samples, rates, percentiles, exemplars, or only minute rollups. Define supported metric kinds: counters, gauges, and histogram-like distributions do not aggregate identically. A counter's rate depends on reset handling and temporality; averaging precomputed percentiles is generally not a valid way to recover a global percentile. Preserve sums and counts or mergeable histogram buckets when later aggregation is required.

Fix out-of-order and duplicate rules. For an append-oriented backend, accept samples within a bounded lateness window and reject older ones with a reason. For cumulative counters, two points with the same timestamp and value may be retransmission, while a lower later value may signal a reset. Do not invent “exactly once” from HTTP. Carry request or batch identity if the product requires replay-safe accounting, and make storage writes idempotent on the chosen sample key.

Quantify Cardinality Before Throughput

Suppose 50,000 hosts each emit 500 metric names with 20 stable label combinations. That is 500 million potential active series before replicas or ephemeral containers. At even 200 bytes of mutable head state, the index alone would approach 100 GB. Adding user_id can multiply cardinality toward the user population and make the estimate meaningless.

Enforce budgets where labels are first materialized. The SDK can limit streams locally, the gateway can apply tenant and metric quotas, and the backend can detect churn. These layers answer different failures. A local limit protects one process; a tenant limit protects shared infrastructure; churn detection finds series that appear briefly and defeat caches. Return actionable errors with the offending metric and label names, sampled safely, rather than logging every rejected value and creating another cardinality incident.

Preserve Aggregation Semantics

Delta points describe change over an interval; cumulative points describe state since a start time. If two collectors both convert the same cumulative series to deltas, a routing change or restart can create false resets. Keep all points for one series on the same stateful aggregator or attach enough start-time and sequence information to detect discontinuity. Rebalancing stateful consumers therefore needs a handoff or a conservative reset rule.

Spatial aggregation combines compatible series across attributes; temporal aggregation rolls one series across time. Dropping instance_id and summing request counts can be correct. Dropping it and averaging host-level p99 latency is not. Store aggregation metadata with the result: window boundaries, retained dimensions, temporality, and completeness watermark. That lets the query layer avoid mixing a five-minute rollup with an incomplete recent minute as if they had equal coverage.

Build a Durable, Partitioned Write Path

A practical path is SDK or agent → regional gateway → durable log → normalization/aggregation workers → hot and cold metric storage. Partition the log by a stable series hash so ordering and stateful delta conversion stay local. Batch across series to improve network and compression, but cap both item count and bytes so one payload cannot monopolize memory.

Authenticate the tenant before expensive parsing, validate timestamps and label sizes, and assign a server receive time. Append the accepted envelope durably before acknowledging if the API promises replay after gateway failure. Consumers checkpoint offsets only after their idempotent storage writes commit. Keep dead-letter handling narrow: malformed data should be rejected at admission, while transient storage failure should remain retryable in the main path with bounded backoff and visible age.

Make Backpressure a Product Behavior

Imagine ingress remains at 4 million samples per second while the database falls to 2.5 million for fifteen minutes. The backlog grows by 1.5 million samples each second, or 1.35 billion samples. A queue with only ten minutes of headroom will overflow. Say which lever activates first: pause optional enrichment, increase batch efficiency, shed low-priority debug metrics, or return 429/503 so clients retry with backoff.

Do not let unlimited in-memory queues hide the problem until processes crash. Give each tenant a fair-share buffer and cap the global write-ahead log. Agents should persist retry state where possible and add jitter, but their local disks also have limits. Define an emergency degradation order before the incident: protect service-level and billing metrics, sample diagnostic series, and reject newly exploding label sets. Every drop needs a counter that is lower-cardinality than the data being dropped.

Design Reads, Retention, and Repair Together

Keep a recent hot tier for dashboards and alerts, compact immutable blocks for longer retention, and precompute only rollups that match frequent queries. Downsampling should retain counter sums, gauge min/max/sum/count, and mergeable histogram state rather than only averages. Use time and tenant partitions that support deletion and bounded scans without creating one tiny file per series.

Reconciliation compares durable log offsets, storage block manifests, and query-visible watermarks. If a worker wrote data but failed before checkpointing, idempotent block or row keys prevent duplicate visibility. If a block is corrupt, rebuild it from retained log data or an earlier raw tier. State the recovery horizon: after raw events expire, a late repair may only reconstruct rollups. That limitation belongs in the service objective, not in an incident footnote.

Choose Where to Aggregate

Client-side aggregation reduces bandwidth and protects the backend, but it loses raw dimensions and trusts heterogeneous SDK behavior. Gateway aggregation centralizes policy and can combine local agents, but it adds state to the admission tier. Stream aggregation scales behind a durable log and supports replay, at the cost of query lag. Storage-time aggregation is flexible but forces the backend to ingest the full cardinality. A strong answer combines them deliberately: safe SDK aggregation, stateless validation at the edge, stateful rollups after the log, and raw retention only where the query contract justifies it.

Compare push and pull without slogans. Pull systems discover targets and control scrape cadence, which is useful inside managed environments. Push is natural for short-lived jobs, mobile devices, and cross-network agents. Either model can feed the same durable ingestion core, but their overload signals differ: a puller records scrape failure, while a push gateway must reject or buffer. Preserve origin timestamps and receive timestamps so queue delay is measurable.

Finally, separate availability from completeness. A dashboard can remain available while showing data only through watermark 12:04. Returning an explicit complete_through value is safer than blending incomplete recent buckets into historical totals. For alerts, decide whether missing data is healthy, unknown, or an alert of its own. That decision often matters more than the storage engine name.

Make the Answer Reviewable

Turn the metrics ingestion system design design into a small deterministic review before adding more scale. Write the initial state, five to ten timestamped inputs, every expected state transition, and the exact read result at two watermarks. Replay one input, deliver another out of order, pause one consumer, and rebuild the serving view from its durable boundary. The replay should not create an extra effect, the stale input should follow the declared version rule, and the rebuilt read should match the authoritative result through the same watermark.

Keep expected results independent of the implementation. If the same code produces both the output and the assertion, a shared assumption can hide the defect. Add one invariant check that scans all state, even if production serves a faster index. For a distributed design, say which part this fixture does not prove: it does not measure production latency, network partitions, or capacity. Load and fault tests come next, using the traffic and error budgets already stated.

Finish with an operational table: signal, healthy range, action, and user-visible consequence. Include ingestion or queue age, rejected or duplicate work, authoritative-versus-serving divergence, and tail read latency. This makes the proposal inspectable. An interviewer can now change one requirement and see exactly which state, test, and alert must change with it.

Separate what the review establishes from what it leaves open. A passing deterministic fixture proves the declared state transitions for those inputs. It does not establish performance at scale, fairness across real users, or behavior during every network failure. Add a capacity test for the estimated peak, a skew test concentrated on one key or partition, and a fault test at the acknowledgment boundary. Record versions and random seeds so failures can be replayed. When a result differs from expectation, reduce it to the smallest trace, name the violated invariant, and repair that transition before retuning capacity. This evidence-first sequence keeps correctness questions from being hidden inside infrastructure changes.

Keep the review artifact beside the implementation and date the environment that produced it. If a later optimization changes ordering, batching, numeric width, or retry behavior, rerun the same trace before accepting the new result. The invariant is the stable comparison point; the component diagram is not.

Practice Five Related PracHub Questions

The first column links directly to five complete PracHub question records. Treat them as drills for transferable reasoning. They are candidate-reported practice evidence, not a forecast of a specific company interview.

A 45-Minute Interview Structure

Use five minutes for metric types, queries, retention, and completeness. Spend eight on series identity and cardinality math. Draw the regional gateway, durable log, stateful aggregation, and hot/cold stores in ten. Use another ten for temporality, duplicates, out-of-order points, and rollups. Spend eight on the overload trace and tenant fairness. Close with four minutes of recovery and observability. If the interviewer asks for more scale, revisit active series and bytes per sample before multiplying partitions.

Final Takeaway

The best metrics-ingestion answer preserves meaning under reduction and overload. Define series identity, temporality, and query completeness first. Bound cardinality before it reaches shared state. Put replayable work behind a durable boundary, and surface backpressure as lag or explicit rejection. Then show that storage blocks, rollups, and dashboards share the same watermarks and repair model.

Sources and Further Reading

• OpenTelemetry Metrics data model
• OpenTelemetry Metrics SDK specification
• Prometheus Remote Write specification
• Prometheus Remote Write tuning

Research note: Technical documentation and linked PracHub practice records were checked on September 13, 2026. Documentation supports the stated system or algorithm behavior; individual question records support only their own practice contracts. Architecture recommendations and interview sequencing are editorial inference.