Feature Store System Design Interview: Freshness, Backfills, and Online Serving

A feature store can answer in five milliseconds and still give a model the wrong information. The value may describe an old observation, belong to a different feature definition, or have been overwritten by a historical backfill. In a feature store system design interview, explain what a value means before explaining where it lives.

For ML and backend engineering candidates, start with a contract connecting entity identity, computation, freshness, history and serving behavior. Then show how that contract survives delayed data and a version change. Practise the architecture with Design a Feature Store for Offline Training and Low-Latency Online Serving on PracHub.

Evidence boundary: Product facts below are attributed to current Feast and AWS documentation. The workload, numbers, failure policies and backfill protocol are original exercises and design recommendations. They are not vendor guarantees or candidate reports. No claim is made about how often an employer asks this topic.

What should you clarify before drawing the feature store?

Use this fictional brief: a payment-risk model needs a customer's transaction count over the previous hour, an account attribute and a merchant statistic. It serves 20,000 predictions per second. Allocate a 15-millisecond feature-retrieval deadline inside the wider inference budget; these numbers are assumptions for discussion, not recommended industry targets.

Ask which features need streaming updates, which tolerate a daily refresh, how many entities are active, and whether requests read one entity or several. Establish fallback consequences: recommendations and payments may need different policies, regardless of database.

Define the transaction count precisely: entity key, event-time window, deduplication key, treatment of reversals and late events, output type, owner and transformation revision. An hourly count calculated from processing time differs from one calculated from transaction occurrence time. Reusing the same column name does not make those definitions compatible.

At 1 KB per bundled read, this traffic implies roughly 20 MB/s of value payload before overhead, replication or retries. Twenty million entities imply roughly 20 GB of values before keys and storage overhead. Explain skew and hot entities before multiplying average capacity into a hardware recommendation.

Separate history, computation and the serving projection

Draw durable source events, batch or streaming transformations, versioned feature history and an online projection. A registry or equivalent catalog records definitions, schemas, owners and model dependencies. Training reads history under a time contract; inference reads a small serving representation under a latency contract.

Official fact: Feast documents an online store containing only the latest feature values for an entity key, rather than historical values. Its materialization process loads feature data into that store. This illustrates why a fast online database cannot, by itself, reconstruct last month's training examples. Feast online store.

Keep source events and feature history recoverable. Save transformation revisions and source progress so a reviewer can distinguish corrected input, changed logic and missing records when reproducing a dataset.

Share a reviewed definition across training and serving where practical, but do not promise parity merely because both paths use one library. Entity normalization, timezone boundaries, null handling, late-event policy and numerical precision can still diverge. Compare outputs on the same recorded entity keys, input events, definition revision and cutoff, including missing values.

How do you measure freshness instead of just setting TTL?

Freshness is the age of the information a model can actually use under a feature-specific policy. A recent publication timestamp does not prove that the underlying observation is recent. Measure source progress, transformation completion and successful online publication separately so an alert identifies the stalled stage.

Here is an original serving exercise. Every timestamp is on the same date in UTC, the request arrives at 12:00, and the count's maximum acceptable age is five minutes. window_end identifies the aggregate's observation endpoint; published_at records successful publication to the serving path. Assume the aggregate calculation itself has already passed its window and late-arrival checks.

At exactly five minutes, this exercise accepts the value; beyond five minutes it rejects it as stale. Future observation timestamps and publication timestamps after the request are invalid for this reference check. Production code also needs clock-skew and timestamp-validation policies.

Official fact: Feast's feature-view TTL limits historical lookback, while SageMaker documents online record expiry relative to EventTime + TtlDuration. These mechanisms do different jobs. Neither statement establishes the payment model's five-minute business rule. Check the selected store's behavior rather than treating TTL as a portable freshness guarantee. Feast feature views, SageMaker TTL.

What should the online serving API return on failure?

Return feature values with enough context to enforce the model's contract: definition or bundle revision, observation time and a status. Keep missing, stale, invalid and transport failure distinguishable in logs and metrics. Preserve these distinctions in operational evidence.

For the fictional payment model, propose routing an unavailable required risk feature to a separately approved fallback model or manual-review policy. An optional merchant statistic might use a training-compatible default plus a missingness indicator. Agree these policies with model and product owners.

Avoid refreshing from the warehouse synchronously after a cache miss. That path can consume the inference deadline and amplify a warehouse incident. Budget downstream calls, bound concurrency and cancel work that can no longer affect the response. A cache of last-known values needs the same age check as the main store.

Bundled records reduce network calls, but grouping everything into one row creates a consistency question: can a reader receive a mixture of revisions? State whether the store supports atomic record replacement and whether the feature bundle requires a shared computation cutoff. A revision tag detects incompatibility; it cannot make independent writes atomic.

Why can online and offline features disagree?

Official fact: Feast push sources support online, offline and combined target modes; the documented default is online. When using a batch source, users are responsible for ensuring data also reaches it. The design implication is that a successful online update is not evidence that historical training data is complete. Feast push sources.

In the proposed architecture, persist replayable events and track each sink's progress independently. Recover a failed historical write by replaying a stable event identity, with deduplication or idempotent replacement defined for that sink. Do not describe two destination writes as one transaction unless the implementation provides that property.

Compare logged production feature vectors against a reconstruction at the same decision time and definition revision. Comparing today's corrected warehouse row with yesterday's prediction confuses an input correction with a serving defect. For the detailed timestamp predicates and SQL, use point-in-time feature join interview questions; here the design question is which evidence the serving path must retain.

How can a backfill avoid overwriting live features?

Suppose transaction reversals were excluded from revision v1. Revision v2 fixes that definition. Historical recomputation should not silently replace the inputs of the still-running v1 model. In this original rollout protocol, v1 and v2 are logical feature namespaces with explicit model bindings, not a promise about a particular platform's versioning feature.

1. Freeze the rebuild inputs. Save the source snapshot, code revision, schema and source-position boundary W. For partitioned logs, W is a set of offsets or equivalent progress markers, not just “12:00.” Preserve enough preceding history to warm up the one-hour aggregate.
2. Build v2 separately. Recompute historical outputs into v2 history and its isolated online namespace. Keep v1 ingestion running. Retries use stable output identities and deterministic revision rules.
3. Catch up and reconcile. Consume changes after W into v2, accounting for the snapshot's exact inclusion boundary. Late events and corrections follow the declared recomputation policy. Reaching W once does not mean the entire event-time history is final.
4. Compare and canary. Validate keys, windows, values, statuses and coverage before routing a compatible model to v2. Shadow reads help measure differences without affecting decisions. A canary tests the model-and-feature bundle together.
5. Cut over with a rollback route. Change an explicit model-to-bundle binding. Keep v1 fresh during the rollback window; an old namespace that stopped ingesting is not a usable fallback.

Consider a delayed backfill output for window 10:00 arriving after a live 11:59 output. Arrival-order replacement would roll the online state backward: the model would suddenly see an older window even though its read succeeded. Define conditional publication using the logical observation order, plus an explicit tie policy for corrections. For mutable same-window aggregates, define how a newer revision replaces an older revision without letting an unrelated historical window win.

Trace one entity: v1 serves count 8 for 11:59; corrected v2 serves 6 for that window. A replay carrying count 99 for 10:00 must replace neither. Model M1 remains bound to v1; only compatible M2 can read v2. Rollback selects M1/v1.

Official product example: SageMaker's PutRecord documentation says a record with greater EventTime is written to the online and offline stores; otherwise it is historical and written only offline, subject to configured targets. That is a provider-specific ordering rule, not evidence that every store protects replay identically. Verify same-time correction behavior rather than assuming it. SageMaker PutRecord.

What evidence makes the cutover reviewable?

A row-count match is weak evidence. Two datasets can contain the same number of rows while assigning counts to different customers. Compare the expected entity-window key set, duplicate identities, null/status rates and values. Separate intentional v1-to-v2 changes from unexpected discrepancies, with examples a reviewer can inspect.

For integer transaction counts, compare exactly. For floating-point features, specify a justified tolerance and report the worst differences rather than rounding everything until it matches. Slice reconciliation by source partition, event period and entity cohort so healthy averages cannot hide an incomplete partition.

Carry run ID, source boundary, definition revision, validation results and approval into the rollout record. Monitor serving latency, errors, feature-age distribution, default usage and model outcomes after cutover. Offline reconciliation cannot establish production performance under load; load and failure tests remain separate work.

How would you investigate a sudden stale-feature incident?

Suppose retrieval latency stays low while stale responses rise for one source partition. Begin with one affected prediction: entity key, requested bundle, observation endpoint, publication time and serving status. Compare that trace with an unaffected entity. Fast database reads suggest that adding online replicas may not address the actual delay.

Check whether new source events reached the durable log, whether the transformation advanced, and whether its output was acknowledged by the online sink. If source progress stopped, investigate ingestion. If computation progressed but publication did not, investigate the sink, retries and rejected writes. A cluster-wide average can conceal one stalled partition, so retain partition-level progress without putting every entity ID into metric labels.

Then check the feature's meaning. A legitimately inactive customer may have no new transactions, yet the rolling count must still decay as old events leave the window. A pipeline that emits only when an event arrives can leave yesterday's count looking current unless expiry or scheduled recomputation is designed deliberately. Observation age alone cannot prove aggregate correctness.

Finally, limit the response to the affected bundle or cohort where possible. Preserve the failing inputs and publication evidence before replaying. After recovery, verify freshness and values, then test the fallback rate; a recovering pipeline can still serve an old cached default. This investigation sequence is a proposed engineering approach, not a report of an actual outage.

Practise the decisions, then defend the trade-offs

These PracHub records are practice material, not predictions of your exact interview. Choose the prompt matching the part of the design you find hardest.

Return to Design a Feature Store for Offline Training and Low-Latency Online Serving. Start with one entity: draw the v1/v2 cutover, then introduce a delayed backfill and a serving timeout. Explain the exact value or fallback the model receives, and the evidence you would inspect before changing traffic.

Sources and Further Reading

• Feast: Online store
• Feast: Feature views
• Feast: Push sources
• Feast: Point-in-time joins
• Amazon SageMaker: PutRecord API
• Amazon SageMaker: Record time to live