Introduction to Distributed Monitoring

Follow two failures through a distributed application: a broken replication path and an unsuccessful browser request. They show why component health and user-visible success need different observations.

Need for monitoring

Monitoring can reveal a failure and its downstream effects early enough to support a response. Preventing a cascade also requires mechanisms such as bounded retries, spare capacity, and recovery procedures; an alert alone does not provide them.

Consider a hypothetical YouTube-like service in which a user uploads intro-to-system-design:

• The UI service (server A) receives the video and passes data to service 2 (server B).

• Service 2 writes to the database and stores the video in blob storage.

• Service 3 (server C) manages replication between database X and database Y.

Suppose service 3 stops replicating while service 2 continues accepting writes into database X. If X then fails and reads switch to Y, a recently acknowledged upload may be missing from Y. The viewer can see “Video not found” even if the video bytes remain in blob storage. This example assumes asynchronous replication and illustrates why replication lag matters before failover.