Design a Telemetry Pipeline That Handles Inconsistent Metric Names from Old Clients

Design a telemetry system for a data infrastructure team. A large fleet of deployed client software reports telemetry: each record names what is being measured in a `telemetry_name` field (for example a request latency, an error count or a feature usage event) and carries a value, a timestamp and some context about the client. Engineers need to query this data for dashboards, investigations and alerts. The interviewer then added a follow-up: many older clients are already deployed in the field and cannot be updated, and they report the same measurements under inconsistent names. The same latency metric might arrive under several spellings and naming conventions depending on the client version. How does your system handle this, and how does it handle schema evolution in general? ### Clarifying Questions - What kinds of clients are these (desktop or mobile apps, services, devices), and roughly how many are active? - How many records per second arrive at peak, and how large is each record? - How fresh must the data be for dashboards and for alerts? - How long must raw data be retained, and is sampling acceptable for high-volume metrics? - Do records contain anything sensitive that must be filtered out or restricted? - Do the inconsistent names differ only in spelling and convention, or can the same name mean different things (or use different units) in different client versions? ### Part 1 — The core telemetry pipeline Design the path from clients to queryable storage: how clients send data, ingestion, buffering, processing, storage, and the query and alerting layers. Give rough capacity estimates using stated assumptions. ```hint Separate collection from use Consider which parts of the system must never lose or block on incoming data, and which parts can be rebuilt later from a durable copy of what arrived. ``` #### What This Part Should Cover - Client-side batching, retries and backpressure, and the ingestion API - A durable buffer that decouples ingestion from processing and allows replay - Storage choices for raw records compared with aggregated, queryable data, and how each is partitioned - Capacity estimates and the dashboard and alerting read paths ### Part 2 — Legacy names and schema evolution Handle the inconsistently named telemetry from clients you cannot change, without breaking existing dashboards, and design how names and schemas evolve from now on so the problem does not recur. ```hint Where should the fix live You cannot change the old clients, so decide at which stage between arrival and query a raw name becomes a canonical one, and what information is needed to make that decision correctly. ``` #### What This Part Should Cover - A canonical metric catalog, and a mapping from raw names to canonical names that is maintained as data - The choice between normalizing at ingestion time and at query time, and the effect on history - Detecting new unknown variants, and handling names whose meaning or unit differs by version - Governance for new clients: registration, versioning and deprecation ### What a Strong Answer Covers - Explicit requirements and estimates that drive the design choices - An ingestion path that stays available and durable under load and client misbehavior - A normalization design that is correct, reversible and auditable, with the raw data preserved - Operational concerns: late and duplicate data, cardinality limits, cost and monitoring of the pipeline itself ### Follow-up Questions - A newly deployed client version starts sending ten times more records than expected. How does the system protect itself? - How would you deduplicate records that a client retried after a timeout? - A mapping rule was wrong for two weeks. How do you correct the historical data and the dashboards built on it? - How would you let teams add new telemetry without a central team becoming a bottleneck?