Design an observability logging system for emitting and querying logs

Design an observability logging system. Services running across a fleet of machines emit log records, and engineers query those records, for example while debugging a production incident. The design must cover both capabilities the interviewer asked for: how logs are emitted and how they are queried. ### Constraints and Clarifications - Scale, retention, query patterns and freshness requirements were not given. Ask, or state your assumptions with numbers. ### Clarifying Questions - How many services and hosts emit logs, and what are the average and peak volumes in records or bytes per second? - Are records structured (fields such as timestamp, service, level and trace ID, plus key-value attributes) or free text? - Which queries must be fast: filtering by service, time range and level, full-text search in messages, lookup of one request or trace ID, or aggregations such as error counts per minute? - How soon after emission must a record be queryable, and how long must logs be retained? Is older data queried less often? - Under overload, may low-severity records be sampled or dropped, or must every record be kept? - Do logs contain sensitive data, and who may query which services' logs? ### Part 1 — Emitting logs Design the path from a log call in application code to durable storage: the client-side API or library, the record format, and the ingestion tier. ```hint The application should not wait Think about what happens to a request being served when the logging backend is slow or unavailable, and where log records wait in the meantime. ``` #### What This Part Should Cover - The logging API and record schema, including the fields every record carries - Buffering and batching in the process or on the host, and the policy when buffers fill - The ingestion tier: authentication, validation, per-service limits, and a durable buffer - Delivery guarantees, duplicates, and how lost records are accounted for ### Part 2 — Querying logs Design how logs are stored and indexed so that engineers can query them, and the query interface they use. ```hint Every query has a time range Use that fact to decide how data is laid out, then decide what else is worth indexing and what can be scanned. ``` ```hint Indexes cost money too Indexing every word of every message makes text search fast but also makes ingestion and storage expensive. Compare that with indexing less and scanning more at query time. ``` #### What This Part Should Cover - Data layout: partitioning by time and service, compressed chunks, and storage tiers by age - The indexing strategy, and its cost at ingest versus its benefit at query time - The query API and its execution: filters, text search, time ordering, pagination, aggregation, parallel fan-out - Protection against expensive queries, and visibility of very recent records ### What a Strong Answer Covers - Capacity estimates (records per second, bytes per day, retained storage) that drive the design - Separation of the write path from the read path, connected by a durable buffer - Explicit choices about loss, duplicates, freshness and retention - Isolation between services, so one noisy service cannot hurt others' ingestion or queries - Monitoring of the logging system itself: ingestion lag, dropped records, query latency ### Follow-up Questions - One service starts logging 100 times its normal volume during an incident. What happens across your system, and how do you contain it? - An engineer needs every log record for one request ID across all services over the past 30 days. How does your design answer that quickly? - How would you add a live tail that streams matching records within seconds of emission? - A user asks for their personal data to be deleted, and it appears in compressed, immutable log chunks. How do you handle it?