Design a **centralized logging system** for a large organization — an internal observability platform (think a self-hosted Splunk / Datadog Logs / ELK) that many teams ship their application and infrastructure logs to. The system should: - **Collect** logs from many applications and hosts written in different languages and running in different environments (VMs, containers, serverless). - Support **near-real-time search and filtering** (e.g. by service, host, time range, and severity), plus free-text search on the message. - Provide configurable **retention policies** (e.g. 7 days hot/fully-searchable, 90 days archived, then deletion). - Handle **high write throughput** with large, bursty spikes (incident storms, retry loops). - Be **reliable** (no or limited data loss) and **multi-tenant** (different teams are isolated, and one noisy team cannot degrade another). Walk through requirements, high-level architecture, the data model and indexing strategy, scaling and partitioning, reliability and delivery semantics, multi-tenancy, cost, and the key trade-offs you'd flag. ```hint Where to start Treat this as a **streaming pipeline**, not a database problem. Sketch the path an event takes — *producer → host agent → collector → durable buffer → processor → store → query* — and reason about which stage must never block the producer versus which stages are allowed to lag. ``` ```hint The keystone component What lets a *slow* indexing path coexist with a *write path that must not block apps*? A **durable, partitioned, replicated log buffer** (Kafka/Pulsar) in the middle decouples them and gives you replay/backpressure instead of data loss. Almost every reliability and scaling answer flows from putting this in the right place. ``` ```hint Storage is two tiers, not one Logs are huge and mostly never read. Resist storing everything in one searchable engine. Think about a small, expensive **hot** tier (inverted-index search, short window) versus a cheap **archive** tier (compressed columnar object storage, queried on demand) — and how *retention* becomes a cheap metadata operation rather than per-record deletes. ``` ```hint Multi-tenancy pitfalls Where does `tenant_id` come from (hint: not the client payload)? How do you stop one tenant's burst from starving others, and how do you lay out indices so you don't either leak across tenants or explode your shard count? ``` ### Constraints & Assumptions State your scale numbers explicitly — they drive every partitioning and storage decision. A reasonable large-org sizing to defend: - **~10,000 hosts**, each emitting ~100 events/sec on average → on the order of **~1M events/sec** baseline, bursting to **~5–10M events/sec** during incidents. - Average serialized event size after enrichment **~1 KB** → roughly **~1 GB/s (~86 TB/day)** raw ingest. - **Ingest latency** target: a few seconds end-to-end from emit to searchable. - **Query latency** target: interactive — sub-second to a few seconds for a filtered search over the hot window. - Logs are **append-only, immutable, time-ordered** events. No per-record updates/deletes; only bulk expiry by retention. - Slight reordering and small windows of duplication are acceptable. **At-least-once** delivery is the realistic target. ### Clarifying Questions to Ask - **Scale & burst profile** — How many hosts/services, what average and peak events/sec, and what burst multiplier should the system absorb without dropping data? - **Search semantics** — Is full-text search on the message body a core requirement, or are structured field filters (service/host/level/time) enough? This decides inverted-index vs. columnar for the hot tier. - **Retention & compliance** — What are the hot/warm/archive windows, are they configurable per tenant, and are there regulatory constraints (PII redaction, immutability, region residency)? - **Tenancy model** — How many tenants, how skewed is their volume (a few whales vs. a long tail), and what isolation guarantee is required (hard isolation vs. logical separation with enforced filters)? - **Consistency expectations** — Is at-least-once with occasional duplicates acceptable, or is exactly-once required? Is bounded ingest lag during an outage acceptable as long as nothing is lost? - **Budget posture** — Is this cost-optimized (logs are mostly write-only) or latency-optimized (long searchable window)? This sets how aggressively to tier and sample. ### What a Strong Answer Covers - **Requirements split** into functional (ingest, search/filter, retention, dashboards/alerting) and non-functional (throughput, reliability, ingest/query latency, cost, isolation), with stated non-goals. - **Back-of-the-envelope sizing** that turns the scale numbers into concrete write volume, hot-store size, and archive size — and uses those numbers to justify sharding and tiering decisions. - **End-to-end architecture** with a clear streaming pipeline and an explicit rationale for the **durable buffer as the decoupling/durability keystone**, plus the responsibility of each stage (agent spool, stateless collectors, processors, hot store, archive, query/API). - **Data model & indexing**: a canonical event schema, time-bucketed indices so retention = dropping an index, field typing (keyword vs. analyzed text), and a field allowlist to prevent mapping/cardinality explosion. - **Scaling & partitioning**: collector autoscaling, the buffer partition-key choice (and its ordering/hot-partition trade-offs), hot-store sharding and lifecycle (hot→warm→frozen→delete), and query-side limits/routing by time range. - **Reliability & delivery semantics**: an explicit at-least-once stance with idempotent writes on a stable `event_id`, plus a concrete walk-through of failure scenarios (agent down, broker loss, search cluster slow, AZ failure, poison messages). - **Multi-tenancy & security**: server-side `tenant_id` resolution, mandatory query-time tenant filter, per-tenant quotas/rate limits, PII redaction, and encryption. - **Cost controls** and **self-observability** (consumer lag as the key leading indicator), and a crisp summary of the defining trade-offs. ### Follow-up Questions - A single tenant suddenly produces 100× their normal volume during an incident. Trace exactly what happens at each stage of your pipeline — and which other tenants are (or aren't) affected, and why. - A team needs to search logs from 60 days ago by free-text on the message. Your hot window is 7 days. How does that query execute, what's the latency, and what could you change to make 60-day free-text search feasible (and at what cost)? - Your search cluster goes down for 3 hours during peak traffic. Walk through what's lost, what's delayed, and what the recovery looks like — what sizing decision determines whether you lose data? - Compare an inverted-index search engine (Elasticsearch/OpenSearch) against a columnar store (ClickHouse) for the **hot** tier. When would you pick each, and how does the choice change if free-text search is/ isn't a core requirement? - How would you prevent a "mapping explosion" from a tenant that logs thousands of distinct high-cardinality JSON keys, without simply refusing their logs?