Setting Up a Machine Learning System
In this lesson8 sections
Setting Up a Machine Learning System
A system design answer should connect the problem, success metrics, architecture, and model lifecycle. This lesson follows those decisions from problem clarification through offline development, online evaluation, and later debugging.
Why this matters in ML interviews
Interviewers rarely ask you to implement a model from scratch. Instead, they expect you to design a scalable ML system for problems such as:
Showing relevant ads in a search engine
Extracting entities (people, locations, organizations) from text
Recommending movies to users
Knowing how a model works does not finish the design. You also need to explain how the surrounding system obtains data, meets its operating constraints, and measures whether the model helps.
Use the following framework to organise those decisions. The later case studies apply it to specific problems and show where the details change.
The ML system design framework
In the following chapters, you will observe that the key steps involved in machine learning project set up are as follows:
Setting up the problem
Start by clarifying the broad prompt with the interviewer. Ask enough questions to agree on the system’s scope and requirements, then state the ML task in a sentence. This gives both of you a shared problem to evaluate as the design develops.
Example: Search engine design
For instance, you may be asked to design a search engine that displays the most relevant results in response to user queries. You could narrow down the problem’s scope by asking the following questions:
Is it a general search engine like Google or Bing or a specialized search engine like Amazon’s products search?
What kind of queries is it expected to answer?
This will allow you to precisely define your ML problem statement as follows:
Build a generic search engine that returns relevant results for queries like “Richard Nixon”, “Programming languages” etc.
Example: Twitter feed ranking
Or, you may be asked to build a system to display a Twitter feed for a user. In this case, you can discuss how the feed is currently displayed and how it can be improved to provide a better experience for the users.
The feed-ranking case study compares a chronological feed with ordering by predicted relevance. A chronological list can bury relevant tweets when a user returns after a gap. This motivates the ranking task below.
After inspecting the problem from all aspects, you can easily narrow it down to a precise machine learning problem statement as follows:
“Given a list of tweets, train an ML model that predicts the probability of engagement of tweets and orders them based on that score.”
Some problems may require you to think about hardware components that could provide input for the machine learning models.
Understanding scale and latency requirements
Another very important part of the problem setup is the discussion about performance and capacity considerations of the system. This conversation will allow you to clearly understand the scale of the system and its requirements.
Let’s look at some examples of the questions you need to ask.
Latency requirements
If you were given the search engine problem, you would ask:
Do we want to return the search result in 100 milliseconds or 500 milliseconds?
Similarly, if you were given the Twitter feed problem, you would ask:
Do we want to return the list of relevant tweets in 300 milliseconds or 400 milliseconds?
Scale of the data
Again, for the search engine problem, you would ask:
How many requests per second do we anticipate to handle?
How many websites exist that we want to enable through this search engine?
If a query has 10 billion matching documents, how many of these would be ranked by our model?
And, for the Twitter feed problem, you would ask:
How many tweets would we have to rank according to relevance for a user at a time?
The performance and capacity lesson connects these workload and latency requirements to the serving design.
The answers to these questions will guide you when you come up with the architecture of the system. Knowing that you need to return results quickly will influence the depth and complexity of your models. Having huge amounts of data to process, you will design the system with scalability in mind. Find more on this in the architecture discussion section.
Defining metrics
Now that you have figured out what machine learning problem you want to solve, the next step is to come up with metrics. Metrics will help you to see if your system is performing well.
Knowing our success criteria helps in understanding the problem and in selecting key architectural components. This is why it's important to discuss metrics early in our design discussions.
Metrics for offline testing
Use offline metrics to compare models during development. For binary classification, the candidates discussed in this course include AUC, log loss, precision, recall, and F1-score. Choose metrics for the problem rather than reporting every metric by default. Search ranking, for example, uses NDCG to assess the ordering of results.
Metrics for online testing
Once you have selected the best performing models offline, you will use online metrics to test them in the production environment. The decision to deploy the newly created model depends on its performance in an online test.
While coming up with online metrics, you may need both component-wise and end-to-end metrics. Consider that you are making a search ranking model to display relevant results for search queries. You may use a component-wise metric such as NDCG to measure the performance of your model online. However, you also need to look at how the system (search engine) is performing with your new model plugged in, for which you can use end-to-end metrics. A commonly used end-to-end metric for this scenario is the users’ engagement and retention rate.
In another scenario, you may be asked to develop the ML system for a task that may be used as a component in other tasks. Again, you need both component level metrics and end-to-end metrics during online testing. For instance, you could be asked to design the ML system for entity linking which is going to be used to improve search relevance.
Here, you will have component-wise metrics to evaluate the performance of the entity linking model individually. You will also be expected to come up with metrics for the search engine where the entity linking component will ultimately be plugged in.
Architecture discussion
The next step is to figure out the architecture of the system. You need to think about the components of the system and how the data will flow through those components.
To get an idea, have a peek at how the architecture of a search engine’s ML system may be designed, below.
The diagram below is a simplified overview. The search-ranking architecture lesson explains the components and their request flow.
Follow the example query "itlian restaurant" through the diagram. Query rewriting corrects it to "italian restaurant". Query understanding identifies the local intent, document selection retrieves relevant candidates from the billions of documents on the web, and the ranker orders those candidates by relevance. The search engine result page (SERP) displays the result. Describing one query this way makes the responsibility of each component concrete.
Architecting for scale
As we mentioned previously, the requirements gathered during problem setup help you in chalking out the architecture. For instance, you are tasked with building an ML system that displays relevant ads to users. During its problem setup, you ask questions and realize that the number of users and ads in the system is huge and ever-increasing. Thus, you need a scalable system that quickly figures out the relevant ads for all users despite the increase in data.
Hence, you can’t just build a complex ML model and run it for all ads in the system because it would take up a lot of time and resources. The solution is to use the funnel approach, where each stage will have fewer ads to process. This way, you can safely use complex models in later stages.
Offline model building and evaluation
The next step is to start building the model offline and then evaluate it. This step involves:
Training data generation: Decide how to obtain enough examples and reliable labels for the supervised task. The collection choices below affect both development cost and what the model can learn:
Human-labeled data: We can hire labelers who can label data for us according to the given ML task. For instance, if we have to perform segmentation of driving images, labelers will use software such as label box to mark the boundaries of different objects in the driving images.
This is an expensive way to gather data. So we need to supplement it with in-house labelers or open-source datasets. “BDD100K: A Large-scale Diverse Driving Video Database” is an example of an open-source dataset that can be used as training data for the segmentation task. It contains segmented data for driving images.
You will see another way to enhance training data in the image segmentation chapter!
The training-data collection lesson compares ways to obtain and expand labeled examples.
Feature engineering: Identify the actors in the task, then describe their properties and interactions. For a Netflix movie-recommendation example, start with the actors shown below:
In order to make features, you would individually inspect these actors and explore their relationships too. For instance, if you individually inspect the logged-in user, you can come up with features such as the user's age, gender, language, etc. Likewise, if you look at the context, an important feature could be the "upcoming holiday". If Christmas is approaching and the movie under consideration is also a Christmas movie, there is a greater chance that the user would watch it. Similarly, we can look at the historical engagement between the user and media to come up with features. An example could be the user's interaction with the movie's genre in the last three months.
The movie-recommendation case study develops the user, content, and context relationships used in this example.
Model training: Choose models that fit the task and its performance and capacity limits, then compare hyperparameter settings. In a funnel, simpler models can process the large initial candidate set; later stages can apply neural networks or tree-based models to fewer candidates. Pretrained state-of-the-art (SOTA) models are another option when transfer learning fits the task.
Offline evaluation: Divide the data into training and validation sets. Train the candidate models, hyperparameter settings, and feature sets on the training data, then compare them on validation data using the metrics chosen earlier. Carry the most promising candidates into online evaluation.
Online model execution and evaluation
Now that you have selected the top-performing models, you will test them in an online environment. Online testing heavily influences the decision to deploy the model. This is where online metrics come into play.
Depending on the type of problem, you may use both component-level and end-to-end metrics. As mentioned before, for the search engine task, the component-wise metric will be NDCG in online testing. However, this alone is not enough. You also need an end-to-end metric, like session success rate, to see if the system’s (search engine’s) performance has increased by using your new search ranking ML model.
If you see a substantial increase in system performance during the online test, you can deploy it on production.
The online experimentation lesson explains how to evaluate a candidate model against the existing system.
Iterative model improvement
Your model may perform well during offline testing, but the same increase in performance may not be observed during an online test. Here, you need to think about debugging the model to find out what exactly is causing this behavior.
Is a particular component not working correctly? Are the features’ distribution different during training and testing time? For instance, a feature called “user’s top five interests” may show a difference in distribution during training and testing, when plotted. This can help you to identify that the routines used to provide the top five user interests were significantly different during training and testing time.
Moreover, after the first version of your model has been built and deployed, you still need to monitor its performance. If the model is not performing as expected, you need to go towards debugging. You may observe a general failure from a decrease in AUC. Or, you may note that the model is failing in particular scenarios. For instance, by analysing the video recording of the self-driving car, you may find out that the image segmentation fails in rushy areas.
The problem areas identified during model debugging will guide you in building successive iterations of your model.
The model debugging and testing lesson provides a process for turning observed failures into the next model or data change.