Telemetry and Metric Specs for a Connected Vending Machine with Recommendations

A team is developing an internet-connected vending machine with a companion app. The machine will recommend products to customers based on their purchase history. The product is at a very early stage: even the hardware has not been built yet. How would you design its telemetry, from deciding what to measure down to the data each machine sends? Assume, as in the reported discussion, that a machine sends its data as reports, each report is a dictionary of fields, and the server stores each report as one row containing information such as the device, the region, the transaction and the product category. ### Clarifying Questions - Are recommendations shown on the machine's screen, in the app, or both? - Can customers buy at the machine without the app, so that some purchases have no known customer? - Will machines always be online, or must they store reports and send them later when connectivity returns? - Which decisions should the first version of the data support: hardware reliability, product demand by location, or recommendation quality? ### Part 1 — Decide what to measure What questions should the telemetry answer, and which metrics would you define first? ```hint Before the hardware exists A field that is missing at launch requires a software update on every machine in the field. Think about which teams will need answers from this data and what each will ask first. ``` #### What This Part Should Cover - Metrics grouped by the decisions they support: machine health, sales and stock, customer engagement, recommendation quality - A small set of headline metrics suited to an early product - What must be captured from the first day because it cannot be reconstructed later ### Part 2 — Events and the report structure Each report is one dictionary that becomes one row. Decide which moments trigger a report, when each report is sent, what a single report represents, and which fields every report and each kind of report must carry. ```hint One row, one moment A purchase passes through several moments, from choosing to paying to the product dropping out. Decide whether a row should describe a whole purchase or one of those moments, and what you lose either way. ``` #### What This Part Should Cover - The meaning of one report, and the event types with their firing conditions - Common identifiers and timestamps, including what deduplication and late uploads require - How rows are stored and turned into tables for analysis ### Part 3 — Metric specifications and attribution Write concrete metric specifications for daily usage and for recommendation performance: name, unit of analysis, numerator and denominator, time window and filters. Explain how a recommendation is linked to a purchase that happens after it. ```hint Carry an identifier forward Think about what the machine could attach to later reports so that a purchase can be traced back to the recommendation the customer saw. ``` #### What This Part Should Cover - Specifications precise enough that two analysts would compute the same number - Time zones, late data and machine downtime in daily metrics - A linking mechanism and attribution rules between recommendations and purchases ### Part 4 — Defining recommendation failure A recommendation was shown, but the customer did not buy the recommended product. Is that a failure? What other kinds of failure can happen, and what additional information must be logged to tell them apart? ```hint Walk the path Trace every step from the recommendation being generated to the product reaching the customer, and ask where each step can go wrong. ``` #### What This Part Should Cover - Whether "shown but not bought" is a failure, and against what baseline - Failure types beyond not buying: system, stock, payment, hardware and model failures - The extra fields needed to separate those cases ### Part 5 — Privacy What privacy risks does this telemetry create, and how would you design for them? ```hint Purchase history is personal Ask what someone could learn about one person from a long record of what they bought, where and when. ``` #### What This Part Should Cover - Which collected fields are sensitive and why - Minimization, consent and separation of identity from telemetry - Retention, access control and deletion ### What a Strong Answer Covers - Starts from the decisions the data must support and works down to events, rather than listing fields first - A clear statement of what one row means, with identifiers that let transactions and recommendations be reconstructed - Metric specifications that are precise about units, windows, time zones and late data - Recommendation success measured against a baseline, with failure types that can be told apart in the data - Privacy designed into the first version ### Follow-up Questions - A machine loses connectivity for two days and then uploads its stored reports. What breaks in your daily metrics, and how does your design handle it? - How would you measure whether recommendations increase sales rather than only predicting what customers would have bought anyway? - A software update changes the meaning of an existing field. How do you keep metrics consistent across versions?