Design of a Distributed Logging Service

Design a pipeline that retains every event selected by the application’s logging policy, without additional sampling in transit. “All events” still needs a defined schema and collection boundary; it does not mean recording every byte of application activity.

Requirements

Let’s list the requirements for designing a distributed logging system:

Functional requirements

• Writing logs: Services within the distributed system must be able to write to the logging system.

• Searchable logs: Users must be able to search logs effortlessly and track the application flow end to end.

• Storing logs: Logs must reside in distributed storage for easy access.

• Centralized logging visualizer: The system must provide a unified view of globally distributed services.

Non-functional requirements

• Low latency: Keep ordinary diagnostic logging off the critical request path using bounded buffering. Specify separately whether a critical audit event must wait for a durable acknowledgment.

• Scalability: The system must handle increasing log volumes and a growing number of concurrent users.

• Availability: The logging system must be highly available.

Building blocks we will use

The design utilizes the following building blocks: