Design Telemetry Ingestion and Reconcile Metric Names

Design a telemetry system that receives metric observations from deployed clients, stores them, and makes them available for analysis. Some existing client versions use different naming variants for the same intended metric. Explain the system design and how it should handle both new observations and historical data without accidentally combining metrics that have different meanings. ### Constraints and Clarifying Questions - Clarify observation volume, arrival delay, query patterns, retention, and acceptable data loss or duplication. No numerical scale is supplied. - Ask how metric identity relates to the producer, tenant or application namespace, units, metric type, and labels. - Assume that existing clients cannot all be upgraded immediately. - A similar-looking name is not sufficient evidence that two metrics have identical semantics. State what evidence and ownership are needed to approve an alias. ### Part 1 — Ingest, Store, and Query Telemetry Propose the path from clients to queryable data. Explain the main design choices and how they change with throughput, freshness, reliability, and query requirements. #### What This Part Should Cover - Client buffering, ingestion validation, durable processing, and storage choices tied to requirements. - Metric identity, label cardinality, and the distinction between raw observations and aggregates. - Recovery, backpressure, and meaningful service measurements. ### Part 2 — Reconcile Naming Variants Explain how to handle a metric that old clients submit under several names. Address incoming data, stored history, dashboards or queries, and a mapping later discovered to be wrong. #### What This Part Should Cover - A controlled mapping from scoped raw names to stable metric identities. - The evidence needed to distinguish a true alias from a different unit, type, or meaning. - A historical-data strategy that preserves provenance and avoids double counting. - A migration, compatibility, and correction plan for existing clients and queries. ```hint A rename can change an aggregate Trace one observation through ingestion, storage, historical reconciliation, and a dashboard. Identify how its identity is preserved and how the system avoids counting both its original and migrated representation. ``` ### What a Strong Answer Covers - A telemetry architecture whose trade-offs are connected to stated requirements. - Consistent name resolution across new ingestion, historical data, and query consumers. - Explicit semantic checks, provenance, and recovery from an incorrect mapping. ### Follow-up Questions - What should happen when an unknown client version submits a familiar name with a different unit? - When would query-time alias resolution be preferable to rewriting historical data? - How would you evaluate a proposed automatic name-matching tool without letting it silently merge live metrics?