Design a Log Collection System
Company: Amazon
Role: Software Engineer
Category: System Design
Difficulty: medium
Interview Round: Technical Screen
Design a scalable log collection system for a company running many services across thousands of machines.
The system should collect application and infrastructure logs, tolerate temporary network or downstream failures, support near-real-time search and filtering by service, host, timestamp, severity, and request ID, and provide retention, alerting, access control, and operational monitoring.
Describe the architecture, data model, ingestion pipeline, storage and indexing strategy, query path, reliability guarantees, scaling approach, and major trade-offs.
Quick Answer: This question evaluates proficiency in designing large-scale distributed logging and observability systems, including ingestion pipelines, storage and indexing strategies, query architectures, reliability and scaling trade-offs, and operational concerns like retention, alerting, and access control.
