Design a log filtering and analytics service
Overview
This question evaluates understanding of distributed systems, real-time data ingestion and enrichment, indexing and storage tiering, query API design, and trade-offs between correctness, latency, and scalability in log analytics.
Amazon
Sep 6, 2025, 12:00 AM
Software Engineer
Technical Screen
System Design
0
0
Design a Log Processing Service
Context
Build a service that ingests application logs from many services and provides low-latency search and analytics.
Functional Requirements
-
Filtering logs by attributes:
- Filter by service, level (INFO/WARN/ERROR), time, and substring/pattern in the message.
-
Error counts:
- Return counts of error-level logs over a specified time window.
-
Histograms:
- Build an hourly histogram for a specific log pattern or pattern ID.
What to Specify
- Ingestion flow (agents, transport, parsing, enrichment, deduplication).
- Storage and indexing (e.g., time-series partitioning, inverted indexes, hot/warm/cold tiers).
-
Query API surface:
- filter(query)
- countErrors(window)
- histogramByHour(query, window)
- Handling late and duplicated events.
- Scalability and partitioning strategy.
- Retention policy.
- Correctness vs. latency trade-offs.
- Schema design with example fields.
- Complexity analysis for common queries.
Solution
Login required
Show