Design a scalable metrics monitoring system

Quick Overview

Design a scalable metrics monitoring system evaluates requirements, scale assumptions, API/data design, architecture, trade-offs, failure modes, and rollout in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Design a scalable metrics monitoring system

Company: LinkedIn

Role: Machine Learning Engineer

Category: System Design

Difficulty: hard

Interview Round: Technical Screen

Design a metrics monitoring system for large-scale services. Compare push vs pull collection models—when to choose each, and their impacts on reliability, backpressure, service discovery, network usage, and failure isolation. Describe the end-to-end architecture: client libraries/agents, ingestion, queueing, streaming aggregation, storage in a time-series database, alerting, dashboards, and SLOs. Propose aggregation and rollup strategies (client-side, agent-side, stream, storage-side), handling of high-cardinality labels, downsampling, late/out-of-order data, retention policies, and backfill. Provide a capacity plan, sharding and replication strategy, and multi-tenant isolation. Explain how you would test and monitor the system itself.

Quick Answer: Design a scalable metrics monitoring system evaluates requirements, scale assumptions, API/data design, architecture, trade-offs, failure modes, and rollout in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

|Home/System Design/LinkedIn
LinkedIn logo
LinkedIn
Jul 16, 2025, 12:00 AM
hardMachine Learning EngineerTechnical ScreenSystem Design
6
0

Design a scalable metrics monitoring system

Design a Metrics Monitoring System for Large-Scale Services

Context

You are designing a metrics monitoring system for large-scale, cloud-native microservices running across multiple regions and clusters. Services are ephemeral (containers/autoscaling), and the platform is multi-tenant (infra teams, ML/feature teams, product services). Assume on the order of tens of thousands of hosts and hundreds of thousands of service instances, with strict SLOs for data freshness and alerting.

Requirements

  1. Compare push vs. pull metrics collection models:
    • When to choose each.
    • Impacts on reliability, backpressure, service discovery, network usage, and failure isolation.
  2. Describe the end-to-end architecture:
    • Client libraries/agents (e.g., SDK or node agent/sidecar).
    • Ingestion layer (APIs, gateways), queueing, and streaming aggregation.
    • Time-series storage, query layer, alerting, dashboards, and SLOs.
  3. Propose aggregation and rollup strategies at each layer:
    • Client-side, agent-side, stream processors, storage-side.
    • Handling high-cardinality labels, downsampling, late/out-of-order data, retention policies, and backfill.
  4. Provide a capacity plan, sharding and replication strategy, and multi-tenant isolation.
  5. Explain how you would test and monitor the monitoring system itself.

Make minimal, explicit assumptions as needed and call out trade-offs and guardrails.

Clarifying Questions to Ask Guidance

  • Clarify users, core use cases, read/write patterns, scale, latency, availability, and data retention.
  • State explicit assumptions before making sizing or architecture decisions.
  • Prioritize the functional path first, then address reliability, security, observability, and rollout.

What a Strong Answer Covers Guidance

  • A scoped requirements summary with concrete non-goals and success metrics.
  • API, data model, architecture, consistency, capacity, and operations.
  • Reasoned trade-offs among simple and scalable designs, including bottlenecks and failure modes.
  • A validation, monitoring, migration, and launch plan appropriate for the risk level.

Follow-up Questions Guidance

  • What breaks first at 10x traffic or data volume?
  • How would you degrade gracefully during dependency failures?
  • What metrics and alerts would prove the design is healthy after launch?

Submit Your Answer to Earn 20XP

Sign in to leave a comment

Loading comments...