Detailed Design of a Monitoring System

Start with one monitoring server, then identify where it runs out of capacity or becomes a failure dependency. The revised design distributes collection while preserving enough detail for useful regional and global queries.

The main choices are storage responsibilities, discovery, collection, rule evaluation, and aggregation. Each needs a defined behavior when its upstream or downstream component is unavailable.

Storage

The proposed design separates three storage responsibilities; they need not imply exactly three physical servers:

• Time-series database (TSDB): Serves recent samples and time-range queries. A local instance is sufficient for the initial design; replication or partitioning may be needed at larger scale.

• Blob storage: Retains older blocks or exports under an explicit lifecycle policy.

• Rules database: Stores alert expressions, evaluation windows, destinations, and configuration versions. A CPU threshold alone is not a complete response policy.

Separating these responsibilities allows recent queries, long-term retention, and configuration changes to have different access and durability requirements.