Design a small-scale metrics monitoring system for database instances in the cloud

Design a monitoring system that collects health and performance metrics from a modest number of database instances, think MongoDB deployments, running on AWS or another cloud provider. It stores the metrics and lets engineers query and chart them. Start by defining which metrics matter and the APIs, then the overall approach. Later, discuss how the system scales and how Kafka and Flink could be combined to make it near real-time. Notifications and alerting are explicitly out of scope. ### Constraints and Clarifications - The fleet is small to begin with; exact instance counts, sampling intervals and retention periods are not given, so ask for them. - Instances may run on more than one cloud provider. - Alerting and notification delivery are out of scope, but the system should still make available the data that alerting would later use. ### Clarifying Questions - Roughly how many instances are there, and how many metrics per instance? - What resolution is needed, seconds or minutes, and how long must raw and aggregated data be kept? - Can we install an agent on the database hosts, or only use the database's own status commands and the cloud provider's metrics APIs? - Who consumes the data: dashboards, an API for other services, ad hoc analysis? - How fresh must the data be on a dashboard? ### Part 1 — Requirements and metrics State the functional and non-functional requirements, and define the metrics you would collect for each database instance. ```hint Three layers Think about host, database-process and replica-set signals separately, and about which values are counters and which are gauges. ``` #### What This Part Should Cover - Functional and non-functional requirements, with alerting kept out of scope - A metric list grouped by layer - How each sample is identified, and the counter versus gauge distinction ### Part 2 — APIs and the small-scale design Define the ingestion and query APIs and the components that collect, store and serve metrics for a small fleet. ```hint Push or pull Decide whether collectors pull from the instances or agents push to you, and what that means across cloud accounts and networks. ``` #### What This Part Should Cover - Registration, ingestion and query APIs - The collection model and its trade-offs - Storage and the query path - Keeping monitoring overhead on the databases low ### Part 3 — Tracking how metrics change over time Engineers want to see how a metric varies over hours or weeks, compare it with an earlier period, and spot unusual change. Explain how storage and queries support this. ```hint Raw is not forever Consider what resolution a month-long chart needs, and what a cumulative counter looks like across a restart. ``` #### What This Part Should Cover - Rollups, downsampling and retention tiers - Rates computed from counters, including reset handling - Period-over-period comparison queries ### Part 4 — Scaling and near real-time with Kafka and Flink The fleet grows and dashboards need near real-time data. Show where Kafka and Flink fit and what each one does. ```hint Who does what Separate durable transport from stateful computation, and decide what is computed in the stream and what at query time. ``` #### What This Part Should Cover - Kafka topic design and partitioning - Flink windowed aggregation and keyed state - Late, duplicate and out-of-order data - Failure recovery and backpressure ### What a Strong Answer Covers - A metric set that reflects how a replicated database actually fails - Clear APIs and a simple first design sized to a small fleet - Correct handling of counters, rollups and retention - A justified path to streaming, including when it is not needed - Gaps shown as missing data, and monitoring of the monitoring pipeline itself ### Follow-up Questions - How do you tell an instance that is down apart from a collector that stopped reporting? - How would the design change for thousands of instances across many customer accounts? - What would you add first once alerting comes into scope? - How do you keep the monitoring system from becoming a source of load on the databases it watches?