Real-Time Late-Delivery Prediction System With Feature Store and Model Registry

In an AI-assisted coding round for a machine learning engineering role, you design and build a working, end-to-end real-time prediction system that runs on your own machine. The round moves fast: you are expected to hand the long prompt to an AI assistant for a summary and to iterate quickly with it. The setting of the prompt: the company runs hundreds of machine learning models at any time to power decisions such as fraud checks, delivery-time estimates and delivery risk. Models change over time, and the data needed to score a live entity, such as an order, a delivery or a customer, is scattered and changes constantly. Support and operations tools need to ask what the model says about a given entity right now and get back a correct, up-to-date prediction, without the caller knowing which model version answers or where its inputs live. Build a system that, given an entity, uses a feature store to assemble the inputs a model needs, uses a model registry to select and fetch the correct model, and runs an inference service that applies the model to the entity's features and returns a prediction. As the example scenario, use a classifier that predicts whether an order will be delivered late. You decide how the order is represented, where its input data comes from, how the right model version is selected, and how a real-time prediction is produced and returned. Requirements and scope stated in the prompt: - Components run independently and communicate through real API calls. - The system keeps running continuously, rather than executing once, so it can be observed live. - You choose the component boundaries, the data model, the API shapes, how many models or entity types to support, and how to handle edge cases. - Out of scope: model architecture, training, and training data. Use a mock model (a function or a trivial classifier is fine) and fake input data. What is evaluated: a working end-to-end system, reasonable scope decisions under time pressure, effective use of AI to speed up implementation, and components that interact as separate systems. Less important: model quality, training, model architecture, perfect abstractions, elaborate class hierarchies and low-level polish. ### Clarifying Questions - How long is the build, and what must be demonstrable at the end: a single request from the command line, or a client calling continuously? - Which languages and frameworks are allowed, and may the components run as separate processes on one machine? - When a feature is missing or stale, should the system return a prediction with defaults, an error, or a flagged answer? - How should a model version be chosen: one current production version per model, routing by entity type, or traffic split between versions? - Does "up to date" mean feature values change while the system runs, so the demo should show a prediction moving as an order's data changes? ### Part 1 — Scope the system and draw its boundaries Decide which components you will build, what each one owns, how an order and its features are represented, and the API each component exposes. State what you will deliberately leave out given the time limit. ```hint Start from the caller Write down the single request a support tool sends and the response it gets back, then ask which component must know each piece of that response. ``` #### What This Part Should Cover - The set of independently running services and the one responsibility of each - The entity and feature data model, and the model metadata the registry stores - Request and response shapes, including what the caller does not need to send ### Part 2 — Assemble features and pick the right model Describe and implement how the inference service turns an entity id into a prediction: how it learns which features the current model needs, how it gets them, how it chooses the model version, and what it does when something is missing, stale or unreachable. ```hint Who knows what the model needs Think about which component should declare a model's required inputs, and what happens to a request in flight when a new version becomes current. ``` #### What This Part Should Cover - Version selection that is hidden from the caller but visible in the response - Feature retrieval, including features that only make sense computed at request time - Behaviour for unknown entities, missing or stale features, and an unreachable registry or feature store ### Part 3 — Keep it running and show it live Make the system run continuously with data that changes over time, and show it working: predictions that change as an order's data changes, and a model version change that takes effect without restarting the inference service. Explain how you used the AI assistant within the time box. ```hint Make change observable Ask what a viewer would need to see in the output to believe that a prediction reflects the current data and the current model. ``` #### What This Part Should Cover - A data simulator and a client that keep traffic flowing - A live model promotion, and a feature change that visibly moves a prediction - How the time was split and how AI-generated code was checked ### What a Strong Answer Covers - A scope stated early and an end-to-end path working before any refinement - Components that are separate processes with real network calls, not functions in one script - A registry that owns model versions and their declared feature lists, so callers never name a version - Responses that carry the model version and feature freshness, so every prediction can be traced - Deliberate handling of missing data and dependency failures - AI used to generate boilerplate quickly, with the candidate owning the design and verifying the output ### Follow-up Questions - How would you promote a new model version while traffic is flowing, and roll it back if its predictions look wrong? - What changes when this serves hundreds of models at production traffic, and how do you keep online features consistent with the ones used in training? - How would you log predictions so they can later be evaluated against actual delivery outcomes? - How would the same inference API serve a different entity type, such as a customer for a fraud check?