Design a metrics monitoring system
Company: Current
Role: Software Engineer
Category: System Design
Difficulty: hard
Interview Round: Technical Screen
Design a metrics monitoring system. Requirements: collect numeric metrics from many services (counter/gauge/histogram) with labels/tags; support pull vs. push ingestion, high throughput, and backpressure; store time-series efficiently with compression and retention tiers (hot vs. cold storage); provide a query language for aggregations, downsampling, and label filtering; generate alerts on thresholds and SLOs with silencing, deduplication, and routing; ensure high availability, horizontal scalability, and multi-tenant isolation; control cardinality growth and enforce quotas; expose dashboards and APIs; discuss data model, sharding, indexing, write/read paths, failure handling, and consistency choices.
Overview: Design a metrics monitoring system evaluates requirements, scale assumptions, API/data design, architecture, trade-offs, failure modes, and rollout in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
Community answers
Answer by siddhantaws
Assumpption
Alert boundry
Last 20 min
Scale
1M request per second
10K servers each server generate 10 metric and there 3 region
stpe would be 10 sec
Functional requirement
Store metric information generated by each server
Raise alert once it meet the criteria
Generate a dashboard for metric for specific series
XOR compression
Rollup step would 1 min after 24 hrs
tenent isolation
Non Functional
Availability >consistency . 99.9 available SLO , SLA 99.99 with error rate
Alert raise p50 -> 10 sec p99 with in 30 sec
Query interval p99 - 600ms
Entity
metric name such as cpu ,memory,disk,4XX, 5XX
series { "metric_name" :"cpu" , "time" : "312313" , "value" : 100, "tags" : {"region ": "us-east" } }
-
Design consideration
pull vs push
PUll
Easy , handle back pressure ,
service discovery , tracking ip , seprate service for fetching data
Push
Each light weight instance need to fetch and call metric with data with queue buffer
it has to handle back pressure once got 429 error or ignore
Drop message
API
GRPC
message MetricMesage {
int id 1 ,
streing metric_name 2,
int64 timestamp 3,
int 32 value 4;
map tags 5;
}
message MetricMesageResponse {
streing metric_name 1,
int status 2,
int delay 3,
}
service SendMetricMessage {
rpc SendMessage(MetricMesage ) return MetricMesageResponse
}
GET /ap1/v1/{tenent_id}/query?metric_name=<>®ion=<>&step=
cursor based pagination
{
"nextpageCustor" :
}
POST api/v1/{tenent_id}/alerts
{
"name" : "cpu_avg_more50",
"notification " : "notification_channel",
"c