Observability platform design: metric ingestion, alert rules and on-call notification
Company: Stripe
Role: Product Manager
Category: System Design
Difficulty: medium
Interview Round: Other
Design a system that watches running software and tells people when something goes wrong: it gathers health signals from services and machines, lets teams see them, and notifies the right person when a problem needs attention. Cover who the system serves and what it must do, how it is built end to end, how it keeps alert noise under control, how it scales, and how you would judge whether it is doing its job.
```hint Work backward from the page
Picture the engineer woken in the middle of the night. Decide what a useful alert must tell them and how rarely it may be wrong, then let that decide what data you collect, how often rules are evaluated and how notifications are routed.
```
### Clarifying Questions
- Is this for internal engineering teams watching the services they own, or a product that lets Stripe's merchants watch the health of their own payments and get notified?
- Which signals are in scope: infrastructure and service metrics only, or also logs, traces and business signals such as the share of payments that succeed?
- How soon after a problem starts must the first alert go out, and through which channels (paging, phone call, chat, email)?
- Roughly how many services and machines will report into it?
### What a Strong Answer Covers
- Users and requirements, including detection speed and the need for monitoring to keep working when the systems it watches fail.
- An end-to-end architecture: collection, ingestion, time-series storage with retention, querying and dashboards, and alert evaluation.
- Alert quality: rule types, deduplication, grouping, routing, escalation and silencing.
- Scale estimates, with control of label cardinality and storage cost.
- Success metrics for the system itself and its main trade-offs.
### Follow-up Questions
- An outage takes down both a core service and the monitoring cluster that shares its infrastructure. How does anyone still find out?
- A team adds a per-customer identifier as a metric label and the number of series grows a thousandfold overnight. What breaks, and how do you prevent it?
- The share of successful payments naturally swings by hour of day and by country. How would you alert on it without paging people every evening?
Overview: Design a system that collects health signals from production software and notifies the right engineer when something breaks. It tests how you set requirements for detection speed, lay out a metrics and alerting architecture end to end, control alert noise and label cardinality, and define success metrics for the monitoring platform itself.