Design a platform that collects GPU telemetry from a large compute environment and exposes both high-level trend analytics and low-level drilldowns. The interviewer gives these requirements: - A collector process must ingest telemetry at both 30-second resolution and 1-second resolution. - Operators need a dashboard showing three-month trends. - Operators must also drill down to per-GPU, per-second details when investigating incidents. - The same analytics environment may later expose a natural-language SQL assistant. The assistant must only query data the user is allowed to access and must avoid dangerous or extremely expensive generated queries. ### Constraints & Assumptions - Treat this as an observability and analytics platform, not a model-training platform. - You may assume standard compute hosts, GPUs, agents, a durable data lake or warehouse, and an internal dashboard. - State any scale assumptions you need, but keep them symbolic or approximate unless the interviewer provides numbers. - Design for correctness, operability, and cost control. ### Clarifying Questions to Ask - How many hosts and GPUs are in scope, and how quickly can that grow? - Which metrics are required: utilization, memory, temperature, errors, job ownership, cluster, workload, or tenant? - How long must 1-second data be retained compared with 30-second aggregates? - Are dashboards mostly internal operations, customer-facing, or compliance-sensitive? - What access-control model already exists for users, teams, tenants, and datasets? ### Part 1 - Collector and Ingestion Path Describe how telemetry should move from GPU hosts into durable storage. Cover collector placement, buffering, retry behavior, schema evolution, deduplication, and how to avoid losing data during host or network failures. <details> <summary>Hint 1</summary> Think about an agent on each host or node, a durable ingestion layer, and separate handling for high-resolution raw data and lower-resolution aggregates. </details> #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### Part 2 - Storage, Aggregation, and Dashboard Serving Design the storage layout and query path for both three-month trends and per-second drilldowns. Explain how you would partition data, precompute aggregates, and keep dashboard queries fast without losing the ability to investigate raw detail. <details> <summary>Hint 2</summary> Long retention and per-second resolution usually need different storage and compaction strategies. </details> #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### Part 3 - Guarded SQL Assistant Extend the design for a natural-language SQL assistant over the analytics data. Explain how to enforce user access and prevent generated SQL from causing incorrect, unsafe, or extremely expensive queries. <details> <summary>Hint 3</summary> Treat the assistant as an untrusted query author. Put policy and query controls outside the model. </details> #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### What a Strong Answer Covers ```premium-lock What a Strong Answer Covers ``` ### Follow-up Questions - How would you detect a missing telemetry gap for one cluster without alerting on every host restart? - What would you change if per-second data had to be retained for a full year? - How would you support both real-time dashboards and offline analysis from the same data? - How would you safely roll out a new GPU metric from the collector to dashboards?