Design log filtering and histogram service

Q: Design log filtering and histogram service

This question evaluates a candidate's ability to design scalable, low-latency real-time log-processing and analytics systems, testing competencies in ingestion APIs, indexing and storage tiering, query and aggregation strategies, deduplication and handling of out-of-order events, and partitioning for scalability, and it falls under the System Design domain. It is commonly asked to assess reasoning about trade-offs between throughput, latency, correctness and operational complexity, and it tests practical architectural design and applied operational considerations rather than purely algorithmic or coding-level detail.

Q: How do I approach System Design interview questions?

System Design questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master system design interviews.

Q: What difficulty level is this interview question?

This is a hard difficulty System Design question, commonly asked during Technical Screen rounds at Amazon.

Q: What role is this question designed for?

This question is commonly asked for Software Engineer candidates at Amazon during technical interviews.

Question

Design a Log-Processing System (Technical Screen)

Context

You are designing a real-time log-processing system from scratch. Logs are emitted by many services as structured events (e.g., JSON) with fields like timestamp, level, component, host, request_id, and message. The system must ingest high throughput, index/store logs for retrieval, and support analytical queries with low latency.

Assume:

Real-time queries over the last 24 hours should return in under 1 second p95 for typical filters.
Sustained ingest could reach 100k events/second (bursty), with occasional out-of-order and duplicate events.
Message pattern filters can be substring or regex.

Requirements

Design a system that provides:

Filtering logs by fields (e.g., level, component, message pattern).
Returning the count of error logs over a time window.
Building an hourly histogram for a specified log predicate.

Specify:

Ingestion API (schema, transport, idempotency, batching, auth).
Storage/index choices (hot vs cold tiers, indexes, compression).
Query interfaces (APIs, semantics, examples).
Aggregation strategies (on-write rollups vs on-read, caching).
Handling of late/out-of-order events and deduplication.
Scalability/partitioning (topics, shards, sort/partition keys).
Retention policies and tiering.
Correctness/performance trade-offs.
Complexity estimates for common queries.

Design log filtering and histogram service

Quick Overview

Design a Log-Processing System (Technical Screen)

Context

Requirements

Solution

Comments (0)