Data quality, lineage, monitoring and backfill design for an ML feature store

Design how a machine-learning platform keeps the data that feeds its models reliable and well governed, with a feature store at the center. The round was a discussion-based design conversation rather than a whiteboard exercise. It walked through a list of data-reliability dimensions (availability, freshness, completeness, validity, consistency, distribution, compliance and lineage, all under the heading of data quality), then turned to the feature store and how to monitor it, and finally to how to run a data backfill. Assume a typical feature store: features are computed from upstream tables and event streams, then written both to an offline store that training jobs read and to an online store that models read at inference time. ### Constraints and Clarifications - Not given: data volumes, the number of features and models, freshness targets, the storage technologies, or which regulations apply. Ask, or state your assumptions. ### Clarifying Questions - Which consumers read the feature store: offline training, online serving, or both, and which matters most? - Are features computed in batch, from streams, or both, and how stale may an online feature value be? - Which data is personal or regulated, and what must happen when a user asks for their data to be deleted? - Who owns each feature and each upstream source, and who is alerted when a check fails? - Should a failed check stop data from being published, or only raise an alert? ### Part 1 — Define and measure the reliability dimensions For each of availability, freshness, completeness, validity, consistency, distribution, compliance and lineage, define what it means for the data behind the models, how you would measure it, and when it counts as a failure. Explain how the checks add up to a data-quality picture that feature owners and model owners can act on. ```hint One check per dimension For every dimension, name the number you would compute on a table or a feature, the condition that makes it a failure, and where in the pipeline the check runs. Make sure no two dimensions end up with the same check. ``` #### What This Part Should Cover - A distinct, precise definition of each dimension, so that, for example, freshness, availability and completeness are not conflated - A measurable check for each dimension and the pipeline stage where it runs - Severity and ownership: which failures block publishing, which only alert, and who responds ### Part 2 — Monitoring the feature store Describe how you would monitor the feature store, covering both the health of the store and its pipelines and the health of the feature values it serves. ```hint Two copies of every feature The offline and online stores are filled by different paths and read by different consumers. Think about how they can drift apart, and how you would notice before a model's predictions do. ``` #### What This Part Should Cover - Health of the serving path and of the jobs that materialize features - Per-feature value health: freshness, missing or default values, drift against a reference - Detecting disagreement between the values used in training and the values served online - Alert routing, and the metadata (owners, expectations) that makes it possible ### Part 3 — Backfilling data Explain how you would run a data backfill, that is, recompute feature values for a past time range: for example, after a feature's definition changes, when a new feature needs history for training, or after an upstream source delivered bad data for some period. ```hint Rewriting the past Ask what a backfilled value should equal at a given past timestamp, which inputs the recomputation may read, and who is reading the data while you rewrite it. ``` #### What This Part Should Cover - Recomputation that is correct as of each past timestamp, without leaking later data - Idempotent, resumable writes and a validated, reversible cut-over - Isolation, so that the backfill neither slows live pipelines nor overwrites fresher online values - Downstream impact: which training sets and models must be regenerated, and how they are found ### What a Strong Answer Covers - Clarifies consumers, freshness targets, regulated data and ownership before designing - Treats the dimensions as measurable checks wired into the pipeline, not as a vocabulary list - Lineage that connects sources, transformations, features, training sets and models, and is used for impact analysis, debugging and compliance - Monitoring that catches silent failures (stale, missing, drifted or skewed features), not only outages and failed jobs - A backfill plan that is reproducible, validated before cut-over, and reversible ### Follow-up Questions - A user asks for their personal data to be deleted. How do you find and remove it in raw data, features, training snapshots and the online store? - A feature's distribution shifts sharply overnight. How do you tell a real change in the world from a pipeline bug, and should the check block publishing? - An upstream table renames a column. Which check catches it first, and what happens to the features that depend on it? - A backfill rewrote several months of one feature. How do you decide which models to retrain, and how do you confirm that the backfilled values agree with what the production pipeline computes for recent days?